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A  sat  of  programs  for  precise  manipulation  of  simple  olanar 
bounded  objects,  by  means  of  visual  feedback,  was  devolaoed  for  use 
In  the  Stanford  hand-eye  system,  The  system  Includes  a  six  .'•agrees 
of  freedom  computer  controlled  manipulator  (arm  and  hand)  and  a  fully 
Instrumented  computer  controlled  television  camera, 

The  Image  of  the  hand  and  manipulated  objects  Is  acquired  by 
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determination  of  thPee  dimensional  positions  from  one  view, 
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LIST  OF  NOTATIONS 


In  the  following  list  E  or  El,  where  I  is  an 
Integer,  stands  for  an  algebraic  expression,  The  following 
"strange"  symbols  are  used  Instead  of  subscripts, 
superscripts  and  other  symbols  which  tre  usually  not  used  In 
line, 

SQRT(E)  -  the  scuare  root  of  E, 

Elt£2  •  El  raised  to  the  power  of  E2, 

ABS(E)  -  the  absolute  value  of  E, 

E1»E2  -  El  mu  1 1 1 P I  led  by  E2, 

E1/F2  -  El  divided  by  E2, 

INVICR3)-  the  Inverse  of  the  matrix  CR], 

The  following  abb  re v j a 1 1 ons  are  used  In  the  report! 

o , c , 3 . “  came(a  coordinate  system 
c,s,  -  coordinate  system 
c  ,  t  •  c ,  camera  table  coordinates 
h.c.s.-  hand  coordinate  system 
h,t*c,-  hand  table  coordinates 
l.c,3,-  Image  coordinate  system 

t. c.s,"  table  coordinate  system 

u. t.c,-  user  table  coordinates 


CHAPTER  1*  INTRODUCTION 
1,1  Background 

The  general  Meld  of  research,  of  which  this  work  is 
e  parti  Is  called  Robotics,  It  rhymes  with  Aeronautics  and 
Electronlos  and  has  the  same  connotations  l,e,  a  field  of 
engineering  complex  enough  to  deserve  Its  own  name,  In  my 
opinion  It  Is  a  natural  outgrowth  of  the  field  of  Systems 
Engineering,  Henoe  I  would  like  to  start  with  a  discussion 
of  systems  in  general  and  then  desorlbe  the  distinct 
features  of  robots, 


We  can  Informally  define  a  system  as  follows:  A 
collection  0f  elements  whjcb  has  a  goal,  The  following 
oomments  w|||  give  more  substance  to  the  definition: 


(a)  Every  system,  especially  a  complex  one,  |s 
constructed  of  sub-systems,  each  with  Its  own  goal  which  Is 
less  Inclusive  than  that  of  the  paPent  system,  If  we 
subdivide  the  system  Into  pieces  whloh  are  small  enough  we 
will  have  devices,  eaoh  having  Its  own  function,  (The 
distinction  bitween  subsystem  and  device  or  between  goal  and 
function  Is  Intuitive  and  not  formal  and  Is  done  to  help  uB 
In  organizing  our  thoughts, 


(b)  We  can  divide  each  system  Into  three  major 
subsystems:  I)  A  sensory  subsystem  which  senses  the  relevant 
variables  In  the  environment  In  which  the  system  I* 
operating,  and  the  Internal  state  of  the  system  Itself,  II) 
A  dec  I s I on-m*k |ng  subsystem  which  combines  the  outout  of  the 
sensory  subsystem,  the  goal  and  stored  Information  to  decide 
whet  to  do  next,  III)  An  effector  subsystem  which  actually 
carries  out  the  actions  soeclflod  by  the  decision-making 
subsystem,  one  of  the  main  asoects  that  make  system  design 
and  ant  lysis  complicated  and  Interesting  Is  the  faot  that 
the  environment  presents  to  th§  system  many  random  and  time 
varying  aliments,  Only  some  of  them,  hopefully  the  most 
Important,  aPe  measured  directly  by  the  sensory  subsystem, 


(c)  During  the  design  phase  and  after,  the  analysis 
of  system  performance  Is  done  by  using  many  tools:  I) 
Mathematical  analysis  of  the  system  model:  H>  Simulation  of 
the  model  on  a  computer!  Ill)  Experiments  with  a  breadboard 
model  IV)  Experiments  with  an  operational  system,  (Tne 
research  reported  here,  for  example,  consisted  mainly  of 
design  and  exotrlmtnts  of  a  breadboard  model,  the  hand-eye 
system  at  Stanford,  which  Is  desorlbed  later), 


(d)  when  the  structure  of  a  system  does  not  Include 
any  human  elements  (It  Is  than  called  an  automatic  system), 
the  goal  that  the  system  oan  acoompllsh  Is  very  specific  end 
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!s  built  v#ry  deep  Into  th®  structure  of  the  system,  An 
alrPlan®  automatic  d I  let,  for  example,  cannot  be  switched  to 
P‘|ot  ships,  The  Principle  of  operation,  the  method  of 
design  and  analysis,  eto,  might  be  very  similar,  but  the 
hardware  ana  the  software  differ  enouoh  to  make  the  switch 
I  mpoas I b  |  e  , 

We  now  orooeed  with  a  definition  of  a  robotl  A  robot 
Is  a  versatile  automatlo  system  In  the  sense  that  It  oan 
aohleve  many  different  (probably  related)  goals  it  the  same 
time  or  at  different  times*  with  no  ohange  or  with  very 
minor  ohange3  In  Its  struoture, 

The  following  two  examples  will  Illustrate  the 
meaning  of  the  above  definition.  These  were  ohosen  slhce  the 
development  of  these  systems  Is  being  aotlve|y  pursued  by 
Industry, 


(a)  Mars  Explorer!  8ecause  of  the  time  delay 
(minimum  of  4  minutes  round  trio),  direct  manual  control  (as 
on  the  Russian  lunar  explorer)  Is  not  practical.  The 
exolorer  should  carry  out  by  Itself  a  variety  of  tasks  given 
to  It  by  Its  oontroller,  For  examplei  navigation,  rock 
ool lection,  so] I  sample  analysis,  The  tasks  are  different 
but  related  In  tho  sense  that  all  Involve  Interaction  with 
the  terrain!  big  rooks  should  fee  avoided,  Interesting  rocks 
should  be  collected,  rooks  should  be  pushed  aside  to  dig  for 
Soil  s«mD|e9,  dlstlnot|ve  roc*<  formation*  oen  he|D  |_ 
navigation,  " 


(b)  Assembly  line  workerl  In  this  oass  manual 
oontrol  w|||  defeat  the  purpose  of  building  one,  The  tasks 
Include!  mating  parts,  Inserting  and  tightening  screws, 
rlvettlng,  welding,  painting,  (20], 


The  vepsptlllty  of  the  robot  demands  versatility  of 
the  three  major  aubsyetems,  Currently  the  most  suitable 
oandldate  f0r  a  versatile  deolslon-maklng  subsystem  Is  a 
general  >urPoss  digital  oomoutsr,  In  the  design  of  the  other 
subsystems  the  temptation  to  build  an  anthropomofphle  system 
le  very  strong,  In  addition  to  the  often  heard  reasons 
oonoernlng  the  optimization  of  mother  nature  over  millions 
of  years,  I  would  like  to  add  two  morel 


(a)  Communication  between  robots  and 
setters  would  be  muoh  easier  If  they  could  share 
exper lenoe, 


the  goal 
sensory 


(b)  Aooommodatloni  In  the  assembly  line  example  the 
ability  of  the  robots  to  move  In  spaces,  to  use  tools,  and 
to  handle  parts,  all  desloned  for  human  use,  would  be  a 
major  asset. 
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The  major  humf.n  sensory  dovlce  Is  the  eye,  For  this 
and  other  reasons  (availability  of  T,V  equipment  for 
example)  equipping  the  robot  systems  now  In  development  with 
vision  sense  Is  universal  practice,  It  Is  coupled  with  an 
arm  and  hand  device  (at  Stanford  C10J#  M.I.T,  U,  of 
Edinburgh  [3],  and  In  Japan*  E.T.L  C 13 3  and  Hitachi  C63),  or 
with  mobility  (at  Stanford  C24]  and  S,R,I  C 1 6  H  >  • 

Some  of  the  problems  encountered  In  +.ne  development 
of  robots  and  which  are  dealt  with  In  the  research  reported 
here  w|||  be  described  In  the  context  of  the  robot  project 
at  Stanford, 

1.2  The  Stanford  hand-eye  system 

The  hardware  of  the  system  consists  of  a  comoutor 
controlled  T,V  camera  and  rneohanlcal  arm,  Both  are  rigidly 
connected  to  a  table  which  Is  the  systems's  work  sDace  (see 
Figure  1),  Over  the  years  a  set  of  programs  have  oeen 
developed  which  do  the  following:  control  the  various 
parameters  of  the  camera,  plan  and  execute  trajectories  of 
the  arm  C173,  analyze  scenes  which  consist  of  several  simple 
planar  bounded  objects  partially  obscuring  each  other 
C193.C73,  determine  colors  and  manipulate  the  objects  to 
chang*  their  position  and  orientation  (Including  stacking), 
(Currently  there  is  work  In  progress  to  analyze  scenes  which 
also  Include  curved  objects  such  as  tools  C13). 

1.3  Visual  feedback  problems 

The  various  tasks  of  the  hand-eye  system  are  carried 
out  In  a  "groS3"  sequential  fashion  as  follows:  First  the 
scene  Is  analyzed  to  determine  the  objects  It  includes  and 
their  position,  orientation  and  color,  Then  the 
manipulations  needed  are  planned  and  executed,  Jf  some  of 
the  parameters  changed  by  the  manipulations  have  to  be  found 
again,  the  scene  analysis  Is  repeated,  The  execution  of  the 
r  "-n  I  pu  let  I  on  Is  not  monitored  hy  the  vision  sensor  (the  Tv 
jamera)  In  any  way,  This  manner  of  operation  Is  in  ortnclple 
relatively  Imprecise,  uneconomical  and  .  sometimes 
"catastrophic",  These  somewhat  strong  allegations  will  De 
explained  next, 

Precision:  We  will  explain  the  Issue  of  precision  by 
an  example,  The  precision  with  which  a  stack  of  cubes  is 
built  Is  determined  by  the  following  parameters! 

(a)  The  precision  of  locating  the  cubes  Initially, 
This  location  is  computed  from  the  location  of  their  Image 
In  the  Image  plane  and  the  mathematical  model  of  the  camera 
that  we  have.  There  are  five  factors  which  affects  this 


3 


< 


1.  CONTINUED, 


* 


nfra?hf'°n  15  T?e  ?ua|l*y  Qf  th®  image,  in  the  orecjslon 
of  the  scene  analyzing  routine  which  located  the  Images  of 

fwiUi5  The  the  mathematical  mod™  of  th. 

camera,  IV)  Th«  precision  of  the  Invariant  parameters  o *  the 

o!!!erib eh  ar!  d#t#pm,n®d  by  a  calibration  program 
JUiIhfl*  at9r’  v  Th!  dUa,,ty  of  measurements  of  the 
angles  parameters  of  the  camera  such  as  the  pan  and  tilt 

JbJhTh!,pr!ei2,0n  of  'nan,eu|®t!on.  Here  we  have  four 
factors  The  first  three  are  similar  to  III»IV»V  above 
pertaining  to  the  arm,  The  fourth  Is  the  precision  of  the 
arm  servo  program. 

to|erate!he  r09U,tant  ®pp0p  may  b®  more  than  we  oan 

hv  whe?  a  scene  !s  a^aly2®d  end  then  changed 

by  manlpulat,ng  some  of  the  objects  it  Irtcludas,  we  know 

and  the  Program  can  know)  fairly  well  hew  the  new  scene 

nJw  ;eJLfle2lr?!  8C!?®  a"alyz®p»  when  called  to  analyze  the 
W0uld  nd*  "»*<•  use  of  this  Information,  A  scene 
anafyzer  that  can  make  use  of  this  Information  and  look  only 

ISleJfl./  3?,nlnfl  dl'f®p®"°®8  whlch  are  caused  by  the 
imprecision  discussed  above,  would  be  much  more  economical, 

?*hf,d“p  ®  =  al'1  the  cube  *ucKln8 
tKanpis,  jf  the  error  la  Iar9e  enough  go  that  the  canter  o' 
gravity  of  the  top  tube  Ft  outaMe  the  top  r*c.C  pf  tn* 

5*8;  «!■%!:* rb*  ?H|  fm" °f*  ***«  ft  t®  rn#.^ 

l*  Ih  Cgtt  !  rt  *****  tt,8t  th®  fcand  sen  be 

moved  away  In  order  *c  simplify  the  fD# 

!S!n!  ! yzap  ’  Any  s°heme  to  avoid  catastrophes  will  ’need 

f=!i? ,B' *5! 11 ‘I  t°,»n«|y*.  sc.n,a  which  Include  th.  h!”. 
Again  the  fairly  good  knowledge  of  the  position  and 
orientation  of  the  hand  could  be|p  In  the  scene  analysis, 

The  natural  solution  to  the  above  deficiencies  Is  to 
use  visual  feedback,  I  cal!  this  solution  "natural"  because 
!}  '■  ®xtensive|y  employed  by  humens,  The  limiting  factors 
of  the  manlpufat.on  errors*  when  visual  feedback  |s  used, 
!"  *?!  resolution  of  the  Image  and.  the  arhi  servo,  Th. 

Bc  1 a?y ,n  this  case  is  Inherently  better  than  for 
open-loop"  operation.  The  problems  generated  by 
the  open-  oop"  mode  of  operation,  and  the  postbfllty  of 
using  visual  feedbaok  as  a  solution  have  been  known*  by 
researchers  In  robotics,  However,  the  1 mp | ementat I cn  of  the 
solution  had  to  wait  for  the  accumulation  of  knowledge  and 
technological  know-hnw  to  reach  the  right  stag.  I  was 
fortunate  to  start  working  with  the  hand-eye  group  ’of  the 
Artificial  intelligence  Project  at  Stanford  when  this  stage 
was  attained,  ,s  w 
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1,4  The  goal  an  short  deaorlotlon  of  the  work. 

The  goal  of  the  research  reoorted  here  was  to  give 
the  hand-eye  system  some  visual  feedback  capabilities  for 
the  tasks  carried  out  by  the  system  at  that  time,  I, a 
achieve  oreclge  grasping#  Placing  and  stacking  of  objects 
(mainly  oubas) , 

The  work  was  divided  Into  three  parts,  devoiooed  in 
approximately  chronological  order#  although  changes  wore 
made  as  the  wopk  In  |atep  parts  advanoedt 

(a)  Development  of  a  simple  scene  analyzer  which  can 
find  lines  and  corners  In  a  small  oart  of  the  Image  frame 
utilizing  as  much  Information  as  is  gi*on  to  It  at  ai| 
levels  of  its  operation.  This  ana|yzar,  which  wa  will  call 
the  oomer-f  Inder,  Is  suitable  for  tne  visual  feedback 
tasks . 

(b)  Development  of  calibration  updating  programs. 
These  programs  are  needed  to  maintain  the  calibrated  mode) 
of  the  camera  at  Its  best  against  deterioration  over  time. 
It  Is  a  general  principle  of  feedback  systems  that  the 
complexity  and  effort  needed  of  the  feedback  part  of  the 
looD  Is  lnVer38ly  Dr°p°rt|ona|  to  the  Dreols'on  that  can  be 
attained  by  the  forward  part  of  tha  |ooo,  In  our  case  this 
relation  Is  not  smooth  since  a  small  change  of  precision 
{•specially  when  the  precision  Is  on  the  same  order  of 
magnitude  as  the  dimensions  of  the  objects  handled)  can  mean 
a  qualitative  change  In  the  nature  of  the  feedback  part  of 
the  loop,  These  programs  make  use  of  the  comer-finder  of 
part  <  a ) ,  The  calibrated  model,  although  partially 
deteriorated,  |s  still  good  enough  so  that  Information 
supplied  by  It  can  be  used  by  the  comer-f  I  nder , 

(c)  development  of  visual  feedback  task  Droorams 
using  the  comer-f  I  nder  of  Dart  (a)  and  relying  on  the 
precision  of  the  forward  oart  of  the  loop  maintained  by  the 
programs  of  part  (b), 

As  an  example  we  w|||  give  here  a  simplified 
description  of  the  stages  of  operation  when  a  cube  (A)  is 
grasped,  with  visual  feedback,  and  thep  stacked  on  another 
cube  (B)  of  the  same  size,  aflaln  with  visual  feedback,  (The 
numbers  after  each  step  refer  to  figure  numbers): 

(a)  The  hand  Is  brought  over  A,  (32(111)). 

(b)  The  fingers  are  closed  tUj  one  of  them  touches 
the  cube  as  detained  by  touch  sensors  on  the  fingers, 

<  32 ( IV) , 

(c)  The  hand  Is  located  In  the  image  using  the 
corner-finder  and  the  error  of  its  position  relative  to  the 


cube  Is  computed, 

(d)  The  fingers  art  opened,  ( 32 ( V ) ) *  and  the  error 
corrected  by  incremental  movement  of  the  «rn  (32CVI ) > .  Steps 
(b)  to  (d)  are  , epeated  till  tne  errors  are  reduced  below  a 
given  threshold#  Then  the  cube  Is  gripped,  (32(VIJl))t 

(e)  The  too  edges  of  +  bottom  cube  are  located  In 
the  Imege, 

(f)  A  Is  brought  above  0,  ( 33 ( I V  >  > . 

<g)  A  Is  placed  on  B  using  force  feedback,  < 35 ( V ) ) , 

(h)  The  hand  Is  located  again  In  the  Image,  and  then 
the  visible  edges  of  the  bottom  face  of  A, 

(I)  The  oosltlon  errors  are  ca|oUlated»  A  Is  lifted, 
(35(VI)>»  and  the  errors  are  corrected  (35(VII),  SteDS  (g) 
to  (l)  ar®  repealed,  till  the  errors  are  reduced  below  a 
given  threshold,  <  35 ( V 1 1 1 ) ) , 

i  y  th*  feedback  loops  * * *  eneCgted  only  tvl«t, 

3m  ;r  i nt#  main  pfublr')  tl>,et  “ *  hid  to  solve  was 
th»  analytli  p*  the  comp  lea  scene,  ccntalnlng  t*o  h*nd  and 
a?  lent  two  ''ore  objectit  to  miaiufi  the  s-*a  I  I  errors  to  be 
oorf*tt"c.  t  3  nanitvi  this  soil  tn»  &<#r  ner-f  |  n  ser  -at 
dajigned,  n.jCn  t#fort  -is  put  into  miintaJnlna  t** 

eccuracy  of  tni  calibration  o'  tne  by  design  I  mg  in# 

calibration  updating  programs  mb  udlnp  other  tcnemss 
deio  r ■ bed  i *i sr . 

Tw0  existing  programs*  the  camera  con\ro||er  and  the 
automatic  focussing  program  were  modified  for  use  In  carts 
(b)  and  (c)  0f  the  work,  <See  figure  2), 

1,5  Related  work 

The  work  described  nere  Is  In  part  a  direct 
put9rowth  of  the  work  don6  at  Stanford  py  I,Sobe|  and  Ideas 
suggested  by  him  C?63,  Parts  are  extensions  of  the  research 
done  at  Stanford  by  J , M , T*nenbaum  C283,  The  work  is  also 
closely  related  and  makes  use  of  programs  developed  by  my 
eo-workers  at  Stanford,  especially  R.Paul  who  developed  the 
arm  trajectory  planning  and  control  and  K.Plngle  who  helped 
develop  and  maintain  the  utility  Programs  and  the  hand-eye 
system  monitor,  (See  Figure  2), 

Two  other  efforts  In  visual  feedback  should  be 
mentioned  herej 

(a)  The  heroic  efforts  of  W.Wlchman  who  five  years 
abo  at  Stanford  tried  to  dtvtloD  visual  feedback  capability 
without  the  support  and  environment  that  we  now  enjoy,  In 
Wlchman's  system  a  general  scene  analyzer  was  used  to 
determine  the  error  of  stacking  two  cubes,  after  the  hand 
was  removed  from  the  scene,  Then  the  hand  was  brought  In 
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■gain  to  correct  the  errors  C29J, 

(b)  work  don*  by  the  Artificial  Intellgence  and 
Robot  group  at  E,T,L  <£ I  ectr eteohn I oa I  Laboratory)  in  Japan, 
The  task  oarrltd  out  Is  the  Insertion  of  a  long  souar®  block 
Into  a  square  hole  C25  3 ,  The  b|ook  Is  long  enough  so  that 
the  hand  holding  It  Is  outside  the  field  of  view,  This 
simplifies  the  scene  analysis. 

1,6  Structure  of  the  report 

Chapter  2  Includes  descriptions  of  the  hardware  used 
and  software  suoport,  In  chapter  3  the  comer-finder  program 
Is  described  In  detail,  ohapter  4  oontalns  the  description 
of  tne  calibration  updating  programs  and  ohapter  5  that  of 
the  various  visual  feedback  tasks,  Chapter  6  summarizes 
the  results  and  suggests  possible  future  developments,  The 
descriptions  of  the  camera  control  and  automatic  fooussing 
programs  are  left  to  appendices,  Chapter  7  desorlbes  some 
general  problems  ooncernlno  the  design  of  robot  systems  In 
particular  and  large  scale  programs  In  general,  These 
problems,  or  examples  of  them,  surfaced  during  the  research. 
Many  of  them  remain  unsolved.  In  some  cases*  comments  on 
the  principles  of  solutions  are  given  with  the  description 
of  the  particular  Implementation  of  these  solutions  in  the 
current  programs, 

Guide  to  the  casual  reader:  Readers  Intorested  In 
the  corner-finder  should  read  Section  2,1,2,  Chapter  3,  and 
Section  6,2,  Readers  Interested  In  calibration  should  read 
Section  2,1,  Chapter  4  and  Section  6,3,  Those  Interested  in 
visual  feedback  and  not  In  the  details  of  how  the  scenes  are 
analyzed,  should  read  Chapter  2,  Sectlpn  4,1,2,  Chapter  5 
and  Section  6.4, 


10 


chapter  25  hand-eye  system  HARDWARE  and  SOFTWARE  SUPPORT 

2.1  Hardware  and  Interface 

2.1.1  Cenera|  notes 

We  have  two  goals  In  t^e  following  hardware 
description,  One  Is  to  gjve  a  general  Idea  of  the  hardware 
Involved  t°  facilitate  the  understanding  of  the  work 
described,  The  other  Is  to  describe  In  detail  the  subsystems 
and  oarameters  which  have  had  a  marked  effect  on  this  work, 

2.1.2  Imaging  system, 

The  Imaging  device  Is  a  commercial  vldlcon  T,v 
camera  with  standard  raster  scan  (525  lines,  60  flelds/soc, 
U2  Interlace),  The  video  signal  has  a  dynamic  range  of  fl-l 
volt  (amplifier  saturation  and  a  special  limiting  circuit 
keens  the  signal  from  being  higher),  The  o-to-o  noise  in  the 
video  signal  Is  approximately  60  mv,  hence  the  signal  to 
nolee  ratio  (maximum  3lgna|  to  r,m,s  noise)  Is  approximately 
6051,  The  video  signal  Is  sampled  at  the  rate  of  333 
samoles/llne  and  256  I Inea/f |e|d.  Only  one  of  the  two 
fields  Is  samoiad,  The  number  of  samp | e s / I  I ne  was  chosen  so 
that  the  vertical  and  horizontal  distances  between  samples 
on  the  vldlcon  fact  Is  eaual.  Sometimes  we  call  the 
sampling  points,  ''cells"  since  the  sampled  values  actually 
correspond  to  the  average  Intensity  absorbed  by  a  small  cell 
on  the  vldlcon  face, 

The  samples  are  converted  to  A  bit  numbers  (0-15), 
packed  9  In  a  word  and  shipped  directly  to  core  memory  via  a 
fast  (24  Mblts/sec)  channel,  The  channel  bit  rate  limits 
the  number  of  blts/samp|e  since  no  local  fast  buffer  is 
used,  The  sampling  resolution  Is  approximately  600/lnch 
or  the  vldlcon  face  which  corresponds  to  1,5  mrads  with  a  l 
inch  focal  length  lens, 

The  digitizing  orocess  Is  flexible  In  the  sense  that 
we  can  specify  to  the  a/D  converter  which  nterval  of  the 
full  video  signal  dynamlo  range  (  lv  >  to  use  as  its  own 
full  range,  The  Interval  Is  specified  In  nultip  as  of  1/q 
volts,  The  location  and  width  of  the  Interval  is  specified 
by  two  parameters  TCLIP  and  BCIIP  which  w|||  be  called  the 
"clips",  (S«e  Figure  3),  By  using  8  contiguous  Intervals 
each  1/8  volts  wide,  we  can  effectively  have  6  bits,  Thls 
Is  close  to  the  maximum  useful  number  of  bits  because  of  the 
s I gna I -to-no } se  ratio  mentioned  abova, 

We  should  note  that  the  vldlcon  camera  was  designed 
with  tolerances  suitable  to  an  adaptable  human  looking  at  a 
monitor  connected  to  the  camera  In  closed  circuit,  Tne 
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tolerances  that  we  would  Ilka  It  to  havt  aa  a  measuring 
davloa  are  much  staffer* 

In  Crd#r  to  conserve  storage  space  we  can  specify 
which  part  of  th*'  total  number  of  256*333  =  35248  samples  to 
store,  This  part  Is  called  a  '’window"  and  is  specified  by 
the  first  and  last  II n •  (FLINE  and  LLINE)  and  the  first  and 
last  sample  on  a  line  (LSIOE  and  RSIDE).  The  window  Is 
always  rectangular  and  parallel  to  the  scanned  reactanQle, 
The  number  of  lines  the  window  Includes  Is  called  Its 
"helflht"  and  the  number  of  samples/llne  Is  called  Its 
"width",  Each  sample  point  can  be  specified  by  Its 
coordinates  in  the  "Image  coordinate  system"  (l,c,s,).  (See 
Figure  4),  The  maximal  size  window,  I . e  the  one  which 
Includes  ajl  the  samples,  is  caljed  the  "frame",  The  4  bit 
number  stored  for  each  point  Is  called  simply  the 
"Intensity"  at  this  oolnt. 

From  time  to  time  we  mention  a|so  "actual  Intensity" 
wMoh  Is  the  Intensity  of  light  reflected  from  the  object  in 
the  direction  of  the  camera,  The  actual  Intensity  is 
determined  by  the  lighting  conditions  and  the  reflective 
properties  of  the  object  and,  to  a  lesser  degree,  the 
reflective  properties  of  the  surrounding  background  and 
other  objects,  This  Intensity  could  be  measured  by  a 
light-meter  situated  neap  the  camera, 

For  a  given  lens  and  filter  (we  have  a  number  of 
each)  the r •  are  two  factors  which  most  affect  the  video 
signal  level,  One  Is  the  aperture  which  Is  not  controlled 
by  the  computer  ana  Is  usually  left  full  open  to  get  the 
maximum  amount  of  light,  Tb«  other  factor  Is  the  vldlcon 
tarflet  voltage,  When  the  automatic  target  voltage  setting 
circuit  (also  oalled  the  automatic  sensitivity  circuit)  Is 
operated  It  changes  the  target  voltage  so  that  the  video 
signal  corresponding  to  the  highlights  of  the  scene  will  be 
Just  below  the  maximum  I  e  v  a  |  (1  vo|t).  When  more  flexibility 
Is  needed,  the  target  voltage  can  be  changed  by  the  computer 
C2qJ,  In  this  oase  we  have  to  take  care  not  to  "burn"  parts 
of  the  vldlcon  face, 

2,1,3  Camera  oontrol 

The  following  or, mere  parameters  can  be  changed  by 
the  computer,  They  all  have  read-out  connected  to  the 
computer  so  that  the  change  Is  controlled, 

(t)  The  lens  used  can  d«  changad  by  rotating  n 
turret  with  4  lenses  of  different  focal  lengths, 
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{ bl  ?h*  a  1 1  ibnct  betwitm  th*  innq  cinmr  tne  fio 
lace  0*  th*  vfdUftn  pen  ts ■  changed,  TM*  is  dam  o >  moving 
tlia  Mfcd  I*  v  1  g  tgon.  th1|  d  I  f tenge  0et>r*l  n*e  the  pingt  it 
*hltn  tnw  camera  1e  focuieid,  TM*  mithpd  &*  fof.uiifng  wa* 
aha»»n  eg  th«t  on*  notninl  i*  I •  needed  fnitim  a*  thi  four 
tut  *oul  d  fa*  neeneo  to  control  the  f  gem*  trig  rfnp  of  non 
llna  Mpenteiy,  A|«9,  the  longer  fgcal  lengt*  Itn*  gin  Be 
feoueeed  it  (Wgrtar  rinDii  than  U  nbrmlly  attained, 

tgj  T^i  pin.  and  tHi  inirit  a*  the  cmiri  m  a* 
changed,  This  produces  ft  translational  as  well  as 
rotational  movement,  (See  Figure  5),  For  examp|ei  when 
using  the  1  Inch  lens,  when  tha  camera  Is  panned  so  that  an 
object  which  appeared  at  the  extreme  left  of  the  frame  will 
now  appear  at  the  extreme  right,  the  lens  center  Is 
translated  by  approximately  2,5  Inches, 

The  pan,  tilt  and  focus  s®rvoes  ara  of  the  sampling, 
on/off  with  threshold  (dead-zona)  typo,  with  constant  rates 
(.06  rad/sec  In  tilt,  ,13  r«d/s«c  In  pan  and  ,012  Inch/sec 
In  vldlcon  movement),  The  sampling  r»ta  Is  60 
samp | es/sec ,  The  thresholds  (full)  correspond  to  1/5 
dapth-of-f I e | d  for  focus  and  3  mrads  In  pan  and  tilt, 

(d)  a  color  fllteri  which  Is  Inserted  between  the 
lens  and  the  vldlcon  face,  can  be  changed  by  rotating  a 
wheel  with  4  filters  (  red,  gr#en,  blue  and  clear), 

2.1.4  Arm  hardwere  and  control 

The  arm  has  6  degrees  of  freedom,  which  neans  that 
within  structural  constraints  It  can  position  the  hand  at 
any  location  In  Its  working  space  and  with  any  orientation 
C233, 


The  control  of  tne  arm  Is  much  more  sophisticated 
than  that  of  the  camera  C173,C18J,  and  will  be  described 
here  very  briefly, 

The  arm  hat  two  rot*ry  Joints  ("shoulder"  Joints), 
followed  by  a  prismatic  Joint,  followed  by  three  more  rotary 
Joints  ("wrist”  Joints),  ending  In  a  hand,  All  Joints  are 
actuated  by  D.C  electric  motors.  The  toraue  generated  by 
each  motor  is  oontrolled  by  varying  the  voltage  oulse  width 
supplied  to  it,  On  each  Joint  there  Is  an  "Infinite" 
resolution  ootent I omater  supplying  position  Information  to 
the  computer  v|a  an  A/0  converter.  On  each  Joint  there  Is 
also  a  brake  to  hold  It  In  Its  position  after  It  stops, 

The  potentiometer  readings  ar*  sampled  60  tlmes/soc, 
The  position  and  computed  veloc'ty  are  compared  to  the 
desired  values  for  this  Instant  of  time,  and  then  the 
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!! QU?  s  .°?mDut®d  for  each  Joint  taking  Into 
account  th.  varying  wa,ght  and  moment  of  Inertia  loading 

Is  bulled1'  At  the  ®nd  °f  th®  traJect0',y  the  remaining  erro? 

T¥5ical  trajectory  execution  takes  less  than  4 
seconds,  The  accuracy  of  positioning  over  the  table  is  ,1  to 
.2  inch.  The  re&eatab I  I  I ty  of  the  ser v0, ( 1  I m I  ted  by  the 
rn?°iUt  Sc  th®  D°t®nt I ometers  and  the  A/D  converter),  is 

-15®  ,hand  ®ons,?tB  of  two  ODDOsIng  flngers  which  are 
ooened  and  c|osed  together  so  that  the  midpoint  between  them 
remains  fixed  relative  to  the  arm.  This  generates  a  sweeping 
nfl  motlon  ,n  the  dlreotlon  of  the  fingers' 

rrovament  • 

.  l  addition  to  Position,  orientation  and 

otha°Den  betW®!?  *he  f'hCers,  the  control  program  olfers 
other  options  which  are  very  useful  In  the  visual  feedback 
tasks  described  in  Chapter  5l 


I  I  the 


Gr*so  I  ng i  The  fingers  are  closed 
opposing  force  exceeds  some  threshold, 

lUK  .  <b>  Placing;  The  arm  Is  moved  straight  down  (with  or 
without  a  grasped  object)  till  the  opposing  force  exceeds 
some  threshold, 

■  i h a  -Th!.  ,n9ida  Dftrt  of  ®ach  finger  Is  protected  by  a 
rubber  cushion  which  serves  also  as  an  antl-sl ippfng  device? 

Each  of  the  cushions  contains  a  touch  sensor,  actually  , 

/  !ch.  0D®n!  ?h*n  th®  r,n0#1,  I*  Pressed  |  ightly 

(■bout  ,1  oz,  is  needed)  against  an  object,  (See  Figure  6) 
We  then  say  that  the  touch  sensor  Is  "activated",  When  the 
touoh  sensors  are  enabled,  the  arm  and  fingers  motions  are 
-  immediately  when  a  f|ngeP  touches  something, 
0therw|3e  the  sensors  readings  are  Ignored,  This  "reflex"  ls 
the  touched  object.  These  touch  sensors  were  devised  by 
v.  Soheinman  to  he|p  |n  the  visual  feedbaok  tasks,  An 

ensemble  of  similar  sensors  situated  on  all  sides  of  the 
fingers  and  at  selected  points  on  the  arm  could  be  used  as  a 
se  |  f-protec  1 1  ng  and  exploring  device  C2], 

„h  $  i  i  b|ack  rectangle  Is  painted  on  the  outside  face  of 
each  finger.  We  w I , I  call  it  the  "hand-mark",  It  help«  in 
the  recognition  of  the  hand  In  the  soene, 
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stages! 


The  operation  of  the  arm  Is  carried  cut  In  tw0 


V. 


a 5.TpaJ«°tory  generations  A  sequence  0f  needed 
movements  ls  planned  and  checked  for  violation  of  Its  own 
structural  constraints,  going  through  the  table,  etc,  Each 
seqment  of  the  trajeotory  Is  planned  under  the  assumption 
that  the  last  segment  will  be  successful  completed  before 
this  segment  will  be  exeouted, 


One  of  the  segments  that  can  be  planned  is  a  "wait" 
segment  In  which  the  arm  oontrol  program  receives  and 
executes  commands  dlreotly  wl thout t go  I ng  through  a  olannlng 
phase,  The  motions  that  can  be  realized  In  this  segment  are 
those  wh  oh  do  not  significantly  ohage  the  petition  of  the 
arm,  This  feature  Is  used  In  the  oorreotlon  phaee  of  the 
visual  feedback  tasks, 


<b> 

trajectory 
happens i 


Trajeotory  execution!  The  segments  of  the 
are  exeouted  In  turn  until  one  of  the  following 


X.' 


(1)  Ths  last  sagment  Is  succassfully  executed, 

J2)  Th#  program  roaches  a  planned  wait  segment. 

(3)  The  program  detects  an  err0r  0r  malfunction  of 
the  hardware, 

(4)  On*  of  the  touob  sensors  Is  aotlvated  (  after 
being  enabled), 


I  n  the 

wait  state  which 


last  two  cases  the  control  program  enters  a 
Is  equivalent  to  a  planned  wait  segment, 


Whan  a  "proceed"  oommand  Is  given  In 
the  execution  of  the  trajectory  Is  resumed 
segment , 


the  wait  state, 
In  the  next 


2,2  Software  environment  and  support 


2,2,1  General  notes 

...  .  principle  the  software  environment  In  yhlch  the 

set  of  Ideas  and  programs  evolved  should  be  Irrelevant, 
since  the  ideas  and  solutions  are  hopefully  general  enough 
to  hevo  scientific  merit  of  their  own,  in  practice  however 
"J*  s°  and  the  Ideas  and  solutions  are  at  least 
influenced  by  the  software  environment,  We  can  look  on  the 

!?li?*r8u,*!]V,r0n!?9r,t  as  ft  Dart  of  th*  flenenl  environment 
within  which  the  hand-eye  system  has  to  operate,  which  also 
Inoludes  suoh  factors  as  lighting  conditions, 
electromagnetic  Interference,  mechanical  vibrations  of  the 
struotures,  etc. 
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Because  of  Its  relevance  a  short  description  of  the 
software  environment  and  support  Is  given  In  the  following 
sect  I ons . 

2.2.2  The  time-sharing  system 

The  Artificial  Inteliaence  Lab  computing  facility 
Includes  the  following  components! 

(a)  Main  processor*  Digital  Equipment  Corps's 
PDP-10/32. 

(b)  Auxiliary  processor*  DEC'S  pDp-6*  which 

handles  the  j/o  devices,  The  hand  and  camera  sery 

programs  are  executed  on  this  processor  In  real  time  and  i 
parallel  to  the  main  program  executed  on  the  main  processor. 
An  Interrupt  clock  Is  connected  to  the  auxiliary  processor 
and  makes  It  possible  to  execute  the  programs  which  are  in 
real  time  mode  at  the  rate  of  62  tlmes/3ec, 

(c)  Fast  core  memory  of  256  Kwords  (36  bit  words). 
The  memory  can  be  accessed  directly  by  both  processors, 
swapping  channel  and  various  other  channels,  The  size  of 
the  user  Programs  Is  ourr«nt|y  limited  by  the  system  to  9f*K, 

(d)  The  system  Is  operated  In  a  time-sharing  mode 
only,  A||  the  Jobs  which  are  active  but  do  not  have  space  in 
the  fast  c0re  are  swapped  to  a  special  disk  (at  a  rate  of 
600  wo  rd/msec ) ,  The  number  of  Jobs  sharing  the  system  are 
rarely  less  than  10  and  frequently  more  than  30, 

(e)  The  back-up  storage  Is  a  set  of  disks  wnere 
flies  are  keDt  for  each  user,  These  files  oan  be  orograms, 
their  oomplled  versions#  dumps  (|,e  core  images),  results 
etc,  Transfer  between  the  discs  and  core  memory  is  2,? 
times  slower  than  swapping, 

(f)  Hard-copy  outout  devicesj  A  line  printer  and  a 
digital  x-y  p  I  otter . 

(9)  The  main  us«r  I/O  device  is  a  keyboard  and  a 
display  monitor.  There  are  two  kinds  of  display  monitors: 
one  Is  a  random  access  CRT  type  and  the  other  Is  a  raster 
scan  TV  monitor  type,  Both  can  display  alphanumeric  text 
and  line  drawlnOs,  The  line  drawings  are  more  pleasing  on 
the  CRT  type  and  vice-versa  for  text,  currently  there  aPe 
six  CRT  type  displays  (Information  International  Inc.  >  and 
sixty-four  Tv  tyD°  displays  (Data  Disc  Inc,).  In  addition  a 
number  of  teletypes  of  various  kinds  are  connected  to  the 
system, 

2.2.3  Languages 

The  high  level  language  used  In  the  programs 
described  here  Is  SAIL  which  is  a  variant  of  ALGOL  with  an 
associative  data  structur8  and  Its  manipulations  added  [27], 
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When  sDtsa  Is  Important  (although  the  compiler  is 
•  f f  lo lent) ,  or  when  Writing  parts  of  Programs  to  be  executed 
on  the  POP-6,  assembly  language  Is  used, 

A  number  of  U I SP  variants*  FORTRAN  IV  and  other 
languages  are  available  In  the  system  but  were  not  used  In 
the  ourrent  work, 

2,2,4  The  Hand-Eye  Monitor  C9] 

Most  of  the  programs  needed  to  run  hand-eye  tasks, 
for  example  the  pan/tllt  calibration  updating  program 
described  In  Chapter  4,  are  too  large  to  fit  Into  the  fast 
memory,  The  above  mentioned  program's  size  Is  130  Kwords, 
nor*  complicated  programs  need  close  to  300  Kwords,  In  order 
to  be  able  to  run,  the  programs  are  broken  Into  parts, 
called  "modules",  that  are  small  enough  to  fit  Into  the 
memory.  Most  of  these  modules  are  general  enough  to  be  used 
In  various  tasks. 

These  modules  are  run  as  pseudo-Jobs  In  the  time 
sharing  system,  Usually  eaoh  Job  has  Its  own  I/O  device,  In 
order  not  to  consume  a  large  number  of  displays  (up  to  8 
ourrently)  and  for  the  oonvenlenoe  of  the  user  all  the 
modules  can  communicate  with  the  user  via  a  single  I/O 
devloe,  Usually  It  Is  a  CRT  type  display  monitor  since 
lln*  drawings  are  heavily  used  as  condensed  and  meaningful 
outputs, 


The  communication  between  the  modules  Is  done  in  tw0 

ways! 


(a)  Sending  "messages”  between  the  modules.  Each 
message  Includes  the  sender  and  Intended  recipient  names, 
the  name  of  the  orooedure  the  sender  asks  the  recipient  to 
execute  and  the  parameters  needed  by  this  procedure, 

(b>  .Altering  the  values  stored  In  memory  cells  In  a 
special  arBa  of  the  memory,  called  the  "world  model",  which 
can  be  aooessed  by  all  modules,  The  variables  in  this  part 
of  the  memory  are  oelled  "global"  variables. 

The  program  that  handles  all  the  communications 
among  user,  modules,  the  time  sharing  system,  I/O  device  and 
the  world  mo^e|  is  called  the  yand-Eye  Monitor.  To  carry  out 
a  specific  task  the  following  modules  are  usually  used?  a 
"driver"  module  which  "knows"  about  the  task,  a  number  of 
utility  modules  <e,g,  the  oorner-f  Inder  described  in  Chapter 
3,  camera  controller,  color-finder,  hard  controller)  and  the 
hand-eye  monitor,  The  construction  of  a  general  purpose 
drlvar  which  would  be  able  to  plan  and  execute  a  family  of 
tasks  Is  a  major  oonoern  and  activity  of  A, I.  research, 
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CHAPTER  3l  CORNER-FINDER  MODULE 
3,t  Introduction 
3,1,1  Genera|  notes 

The  purpose  of  the  corner-finder  Is  to  find  lines 
and  corners  (which  are  the  main  features  of  Dlanar  bounded 
objects)  In  a  small  area  of  the  frame,  utilizing  Information 
about  the  features  to  the  extent  given  to  It.  As  3uch,  It  Is 
not  a  general  scene  ana|yzer  (even  In  the  context  of  planar 
bounded  objects), and  although  It  can  be  used  as  Dart  of  one. 
It  will  be  uneconomical  to  do  so,  The  cornor-f I nder 
operates  by  analyzing  part  of  the  area  (a  window)  at  a  time 
and  moving  the  analyzed  window  In  a  controlled  search 
pattern  when  needed, 

Most  of  the  soene  analyzers  used  In  robot  projects 
operate  In  two  st^Dsi  (a)  Extract  a  line  drawing  from  vhe 
Image,  (b)  Analyze  the  line  drawing,  The  lines  are 
aotually  boundaries  between  Image  regions  of  difforent 
characteristics,  Two  main  types  of  scene  analyzers  uslnq 
simple  Intensity  Information  have  been  developed  over  the 
years  * 

(a)  The  "gradient  follower"  type  looks  for 
boundaries  of  re9l°ns  by  analyzing  Intensity  gradients  at 
Image  points,  Boundary  points  are  points  with  relatively 
larOe  gradients,  Then  a  direction  perpendicular  to  the 
gradient  Is  followed  to  get  the  next  point  to  be  analyzed, 
The  advantage  of  this  scene  analyzer  Is  that  It  does  not 
have  to  process  all  the  oolnts  In  the  frame,  It  falls  at 
points  where  the  gradients  are  weak,  at  very  sharp  corners 
or  when  two  lines  come  close  together  C123»C19D, 

(b)  Tne  "region  grower”  type  agqregatos  points  based 
on  some  similarity  criterion  to  form  regions,  Rules  are  then 
used  to  merge  regions  Into  bigger  regions,  Boundary  points 
are  defined  simply  as  points  adjacent  to  two  regions,  The 
disadvantage  of  this  type  Is  that  every  point  in  the  frame 
Is  processed  several  times  C43, 

The  corner-finder  uses  Ideas  from  both  these  types. 
It  makes  rough  checks  on  the  existence  of  regions  In  the 
analyzed  area.  For  this  puroose  each  point  within  the  area 
Is  processed  simply  to  form  the  Intensify  histogram  of(  the 
area,  Thei  It  follows  boundaries  of  regions  by  using  a 
dissimilarity  criterion,  No  gradient  type  processing  Is  used 
so  that  continuity  Is  not  lost  at  points  of  weak  gradient, 
sharp  comer,  etc,  General  scene  analyzers  do  not  use  any 
prior  information  because  there  is  no  reason  for  them  to 
assume  the  existence  of  such  Information,  On  the  othPr  hand 
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the  oorner-fj  ,-r  described  here  use*  prior  Information  down 
to  lt«  lowest  I ov » | • .  It  oan  uee  at  much  Information  as 
glvjn  to  It  but  dots  not  rtaulr*  complttt  Information  fop 
proper  functioning,  The  dttlgn  philosophy  Is  to  use  and 
chtoh  against  prior  Information  at  tht  tarllsst  oesslbit 
moment , 


These  Introductory  remarks  are  followed  by  the 
definition  of  terms  used  throughout  and  a  more  detailed 
description  of  the  underlying  assumptions  and  resign 
philosophy,  Then  we  present  the  details  of  the  workings  of 
the  analyzer  (FINQIMAG)  and  the  search  controller 
(SRCHIMAG),  Last  we  detOrlbe  the  parameters  controlled  by 
the  user,  the  format  of  the  message  Procedure  end  program 
output. 


3,1,2,  Definitions  of  some  terms 

The  t«rm  "corner”  Is  used  to  describe  a  number  of 
r e | axed  ent I t I es  : 

(a)  A  corner  of  a  physical  Planar  bounded  object. 
Including  the  vertex,  the  edges  and  the  faces  associated 
with  this  corner,  The  "location”  of  the  corner  refers  to  the 
location  of  the  vertex  given  by  Its  three  coordinates  In  a 
coordinate  system  tied  to  the  table,  (See  Figure  7(a)). 


(b)  The  Image  of  the  ohysloal  oorner  as  encoded  In 
the  stored  Intensities,  (See  Figure  7(b)), 

(c)  An  approximate  line  drawing  of  the  Image  which 
Is  the  output  of  the  oorner-f I nder  program,  The  line 
segmente  emanating  from  the  vertex  and  truncated  by  the 
window  perimeter  are  called  again  "edgee"  of  the  corner. 
The  "location”  of  the  corner  refers  to  the  location  of  the 
vertex  of  the  Mne  drawing  given  by  Its  two  coordinates  In 
the  Image  coordinate  system,  We  distinguish  between  tw0 
kinds  of  oorners  I  "simple"  corners  whloh  have  only  two 
edges,  whloh  will  be  called  "sides",  and  "oomolex"  corners 
which  have  m0re  than  two  edges,  (See  Figure  7(c)  and  7(d)). 
The  corner-finder  oan  find  only  simple  oorners  dlreotly, 
Comdex  comers  can  then  be  constructed  from  simpler 
corners,  Generally,  the  vertloes  and  adgea  cf  simple 
corners  found  |n  the  Image  will  not  completely  coincide  even 
If  the  simple  corners  are  carts  of  Lhs  aama  complex  corner, 
Therefore  we  w]  1 1  merge  them  to  form  a  oomplex  comer  If 
they  are  ”c|ose"  (within  some  tolernnoe),  and  especially  if 
there  Is  some  external  Information  whloh  Indicates  the 
exlstenoe  of  a  oomplex  corner  rather  than  that  of  several 
separate  simple  corners,  (See  Figure  7(a)), 
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From  here  on  the  word  oorner  by  Itself  w|  |  |  refer  to 
a  simple  oorn*r  of  the  line  drawing, 

The  "Inside”  of  •  corner  Is  the  region  of  the  Image 
bounded  by  the  two  sides  of  a  corner  end  Including  the  angle 
between  the  sides  whloh  subtends  less  than  »  reds.  In  an 
Image  of  a  planar  bounded  convex  body  (not  obscuring  or 
obsoured  by  other  oodles)  the  Inside  of  any  corner  is  the 
Image  of  a  cart  of  the  body  and  the  outside  Is  the  Image  of 
the  surrounding  background. 


The  s'de  which  Is  In  a  counter-clockwise 
from  any  point  Inside  the  corner  Is  ca.led  the 
and  the  other  Is  oalled  the  "second”  side, 
7(d)) . 


direction 
first”  side 
(See  Figure 


When  the  Image  Includes  edges  only  and  no  corners 
the  line  drawing  will  consist  of  separate  lines  only,  rfe 
call  a  line  or  oorner  In  the  line  drawing  a  "feature”, 


We  define  the  "form"  of  a  feature  as  follows:  f  j 
Is  a  line*  then  the  form  Is  the  orientation  of  the  line  in 
the  Image,  Jf  It  Is  a  corner#  than  the  form  Is  the 
or  I  epitst  I  on  of  both  sides  a3  given  for  example  by  two  pairs 
of  direction  cosines, 


3,1,3  Basic  assumptions  and  design  philosophy 


The  following  assumptions  guided  the  development  of 
the  corner-finder.  They  are  net  all  neoessary  conditions 

for  Its  operation  or  suooass, 


The  most  Important  assumption  Is  that  some  of  the 
properties  of  the  oorner  <«.fl  location,  form  and 
orientation#  relative  Inside  to  outside  Intensity)  are  known 
at  least  apor ox  I  mate  I y ,  They  are  known  either  because  the 
properties  of  the  object  whloh  this  oorner  belongs  to  are 
known  (for  example:  the  hand  or  a  specif. c  cube),  or 
because  this  oorner  was  found  before  by  the  same  or  similar 
programs,  The  reasons  that  these  properties  am  known 
sometimes  on|y  approximately  are  of  the  following  flavours: 


(a)  Incomplete  coordination  between  the  hand  and  the 
eye,  as  for  example  when  looking  at  the  hand  or  at  an  object 
whloh  was  moved  by  the  hand.  This  problem  and  some 
solutions  are  dlsoussed  In  Chapters  4  and  5, 


(b)  The  pan/tllt  need  of  the  camera  moved,  and  we 
oannot  Predict  «ccurate|y  the  new  location  of  the  comer  in 
the  Image  because  Its  range  from  the  camera  js  not  known  and 
the  camera  head  movement  causes  translation  In  addition  to 
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(b) 


(c) 


(a)  Corner  of  an  object 

(b)  Intensity  map  of  the  image  of  the  corner 

(c)  Line  drawing  of  the  corner  (complex  corner) 

Fig  7- 

ILLUSTRATION  OF  THE  DIFFERENT  MEANINGS  OF  THE  TERM:  CORNER 
(Continued  on  next  page) 
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Simple  Corner 

Construction  of  a  complex  corner  from  two  simple  corners. 


Fig.  7- 

ILLUSTRATION  OF  THE  DIFFERENT  MEANINGS  OF  THE  TERM:  CORNER 

(Continue  3) 


rotation  because  the  tons  oenter  does  not  lie  on  elthr  th* 
D»n  or  tilt  axes, 

Not  a | 1  the  orooartlas  have  to  ba  g'ven  to  tn« 
proflram,  Tha  uyar  or  a  higher  level  program  oan  glva  as  many 
0/  tha  orooartlas  as  ha  (It)  daoldas  to  alva,  Actually  tha 
orooartlas  ara  not  only  "g|v*n"  to  tha  program#  but  tha  user 
oan  "demand"  «  Matoh#  within  a  fllvan  tciaranoc#  of  these 
properties  and  tha  eotuai  measured  properties  of  the  corner 
found# 


Tha  ourr*nt  version  of  tha  program  rejects  r>ny 
corner  which  does  not  matoh  all  the  properties  demandod  to 
ba  matched#  Another  or  future  version  might  give  as  an 
answer  the  oorn*r  whloh  matohee  best#  according  to  some 
given  criterion,  This  version  pould  take  more  processing 
time  and  would  beoome  rather  oomplex  when  the  corner  Is 
senrohed  for  by  moving  a  window  In  the  frame#  Using  the 
first  version  Instead  of  the  ssoond  saves  processing  time 
(at  the  ooet  of  flexibility  and  oenerillty)  at  leagt  In  the 
three  fc I  lowing  waysi 

(a)  There  Is  no  need  to  compute  the  criterion  and 
make  the  comparison  for  each  corner  found, 

(0)  After  one  or  tb«  required  number  of  oornars 
whloh  matoh  property  have  been  found#  there  Is  no  need  to 
look  for  more  oorners  whloh  might  have  <1  better  match, 

(c>  a  oorner  oan  be  r*Jeot«d  Immediately  after  the 
first  mls-matoh.  Is  dlsocversd,  This  Is  done  by  trying  to 
rejeot  the  corner  after  each  stage  of  the  computation  using 
ali  the  knowledge  already  available  at  this  stage, 

The  principle  mentioned  In  (o>  is  also  spoiled  in 
the  Initial  stages  of  the  orooasslng  where  tha  program  is 
trying  to  r*Jaot  the  hypothesis  that  there  Is  any  corner  at 
all  In  the  analyzed  window,  For  a  trivial  example#  when  It 
Is  found  that  the  Intensity  Is  almost  uniform  over  the 
window#  the  hypothsls  Is  rejected, 

A  second  assumption#  whloh  Is  a  necessary 
oondltlontls  that  eaoh  region#  defined  as  a  contlgugus  part 
of  the  analyzed  window  bounded  either  by  the  perimeter  or  by 
edges#  will  have  almost  uniform  Intensity  which  Is  distinct 
from  those  of  neighbouring  regions  In  the  following  sensei 
In  the  Intensity  histogram  there  Is  at  least  one  noticeable 
(see  description  of  the  block  SLICE  In  Section  3,2,2) 
minimum  between  the  Intenelty  levels,  (see  Figure  8),  The 
Intensity  Is  not  oomolstsly  uniform  for  the  fcljowlng 
reasonsi  slight  differences  In  the  reflecting  Properties  of 
the  surface  and  lighting#  noise  In  the  oamera  and 
quantization  of  an  Intermediate  Intensity  level  Into  tw0 
sdjaoant  Quantization  |svs|s  by  ths  A/0  oonvsrter,  This 
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CASE  A 


Cages  A  and  B  -  Histograms  with  noticeable  minima 

(1)  Hypothetical  intensity  of  region  1. 

(2)  Hypothetical  intensity  histogram  of  region  2 

(3)  Combined  intensity  histogram  of  the  window. 

Fig.  8. 

EXAMPI.ES  OF  HISTOGRAMS 
(Continued  on  next  page) 


CASE  C  CASE  D 

Cases  C  and  D  -  Histograms  with  non-noticeable  minima 

(1)  Hypothetical  intensity  histogram  of  region  1. 

(2)  Hypothetical  intensity  histogram  of  regions  2. 

(3)  Combined  intensity  histogram  of  the  window. 

Fig.  8. 

EXAMPLES  OF  HISTOGRAMS 
(Continued) 
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assumption  lnf|uenoes  the  choloe  of  window  size,  (See 
Figure  9),  In  some  cases  the  presence  of  a  simple  corner, 
whjoh  Is  a  part  of  *  oomplex  corner,  can  be  Inferred  from 
other  s I mp I s  oorntrs  found  In  the  window,  See  Figure  12  for 
an  example,  This  assumption  also  Influences  the  choice  of 
window  size,  In  most  soene  analyzers  that  look  for  polygons 
In  the  Image,  straight  line  segmtnts  aPe  first  fitted  to  the 
set  of  boundary  points*  From  these  line  segments  the 
presence  and  looatlons  of  oorners  (simple  and  complex)  are 
computed,  Because  of  the  above  assumption*  the  corner-f I nder 
will  try  to  fit  either  one  line  segment  or  two  connected 
line  segments  (!*•»  «  elmple  eorner)  directly  to  a  set  of 
contiguous  boundary  points  found, 


A  fourth  assumption  Is  that  the  Image  of  the  corner 
oocup las  more  than  a  oertaln  fraction  of  the  window  area, 
When  a  small  refllon  Is  found  It  Is  rejected  because  It  Is 
assumed  that  either  It  Is  a  nclsy  patch  or  that  It  is  an 
honest-to-Cod  oorner  but  that  we  do  not  see  enough  of  it  and 
that  It  w|||  be  subsequently  found  again  when  the  window  Is 
moved  around  In  the  frame  |n  search  of  the  missing  corner, 

We  distinguish  between  two  kinds  of  scenes:  One  Is 
relatively  simple*  for  example  when  looking  at  the  hand 
alone*  (see  Figure  28),  The  other  Is  cluttered,  for  example 
when  looking  at  two  stackeci  cubes,  the  top  one  still  grasped 
by  the  hand,  <see  Flour®  35<V)f.  When  we  are  looking  for 
more  than  on#  feature  In  ths  window,  we  Implicitly  assume 
that  the  scene  Is  cluttered,  In  this  case  we  do  not  like  to 
staroh  with  the  simple  minded  search  strategies  described 
later  In  Section  3,2,3  and  leave  It  to  the  user  or  higher 
level  programs  to  tailor  thalr  own  search  strategies  to  the 
cluttered  scenes  they  expect, 

Some  comments  about  window  s I ze :  The  wl ndow  s I ze 
which  is  regularly  used  has  a  dimension  of  18*18  raster 
units,  When  the  2  'neh  focal  length  lens  Is  used  .t 
corresponds  to  a  field  of  view  of  approximately  1  degree 
whloh  Incidentally  le  the  field  of  view  of  the  sensitive 
part  of  the  human  «y®,  the  fovea,  The  fovea  however  has 
about  5  times  more  sensing  elements  In  the  same  field  of 
view  C53.  I’e  should  note  howevsr  the  human  ability  to 
resolve  between  pairs  of  lines  that  ar#  'loser  than  the 
distance  between  the  sensing  elements,  Carrying  the  above 
analogy  a  Htt|e  farther  we  oan  say  that  moving  the  window 
In  tha  frame  Is  similar  to  the  movement  of  the  eye  in  the 
head,  while  moving  the  camePa  !g  slw'!»r  to  rotating  the 
head, 


Thi-  size  of  the  window  was  chosen  In  order  to 
fulfill  the  assumptions  that  the  window  Is  smaller  than  the 
Image  of  the  object  so  that  each  line  or  corner  Intersects 
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CASE  A  CASE  B 


the  perimeter  of  the  window,  but  big  enough  so  that  we  will 
have  nnouoh  boundary  points  to  get  a  good  line  f?t,  Also  we 
want  the  size  of  the  window  to  be  small  enough  sc  that  the 
assumption  of  almost  uniform  Intensity  Inside  the  window  is 
just  I  fled. 

In  Our  systsm,  when  we  want  to  read  In  a  window,  we 
have  to  wait  uo  to  1/30  seo,  till  this  window  Is  scanned 
again,  Hence  we  might  save  time  by  taking  In  the  part  of  the 
frame  we  want  to  search  all  at  one  time  and  then  run  on  the 
stored  Image,  Ther®  are  two  disadvantages:  First  In  memory 
size  and  the  number  of  memory  cyoles,  and  second,  we  lose 
the  flexibility  of  ohooslng  the  clip  levels  for  each  window 
I ndependent I y , 


3,2  Program  structure 

3.2.1  General  notes 

We  start  with  the  description  of  FINDIMAG  because 
Its  details  are  needed  to  understand  the  description  of 
SRCHIMAG  which  oalls  F I  NO  I  MAG  at  various  stages,  However, 
r I  NO  I  MAG  Is  not  directly  callable  by  the  i  -r,  (or  other 
modu  I  es) , 

3.2.2  Structure  of  FINDIMAG 

The  biock  diagram  of  FINDIMAG  Is  shown  In  Figure  11. 
The  names  cf  the  blocks  are  by  no  means  exact  descriptions 
of  them,  Not  all  the  blocks,  and  not  only  blocks  are 
distinct  orocedures  ll  the  orogrammlng  language  sense,  The 
descriptions  of  the  blocks  f  o  I  low* 

OPENCLIP 

The  cilD  lavels  are  set  so  that  the  maximum  dynamic 
range  Is  digitized. 


INPUT 

All  the  parameters  (camera  number,  window  location, 
window  size  and  cllo  levels)  «(*e  checked  for  trielr  legality 
and  then  the  window  Is  read  once  Into  store,  if  any  of 
the  parameters  Is  Illegal  INPUT  exits. 


H I STO 

The  Intensity  histogram  i.e  for  each  intensity  level 
<0  to  15),  the  number  of  samples  within  the  window  which 
have  this  Intensity  Is  found, 


SETCLIP  _ 
The 

buckets  tv 


c|lp  levels  are  set  so  that  the  dynamlc  range 
actual  range  of  Intensity  levels  found  In 
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START 


OPEN  CLIP 


INPUT 


HiSTO 


SETCLIP 


INPUT 


SLICE 


TART CORNER 


PERIMETER 


FOLLOW 


PRUNE 


CORNFIT 


RELINT 


RECENTER 


MORE 


Fig.  11. 

BLOCK-DIAGRAM  OF  FINDIMAG 


» 


HISTO,  (See  Figures  3  and  12),  This  Insures  that  the  full 
t  dynanilo  range  of  the  A/D  oonverter  Is  utilized  without 

losing  any  Ipformatlon,  This  Is  Important  since  the  w'.ole 

operation  of  the  oorner»f I nder  Is  based  on  finding  two  or 

more  maxima  In  the  histogram, 

The  computation  of  the  new  clip  levels  is  as 
t  follows*  If  MAXI  and  MINI  denote  the  maximum  and  mlnlmun 

Intensities  In  the  window  then 

TCLIP*(15-MAXI )/2  and  BCL I P® < 15-M I N I )/2, 

The  clip  levels  can  be  closed  even  further  by 

*>  snipping  the  tails  off  the  histogram,  This  will  utilize  the 

dynamic  range  even  better  without  losing  Important 

Information,  (See  Figure  13), 


f 


S 


z 


z 
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HISTO  Is  used  to  form  the  Intensity  histogram  (which 
we  will  call  h ( I ) ) ,  Then  the  maxima  of  the  histogram  are 
found  as  follows*  There  Is  a  maximum  at  Intensity  I  If 
H ( I )  Is  |arger  than  a  glvgn  threshold,  H(I)>H(I-1)  and 
H < I ) >H ( I *1 ) ,  (H(16)  Is  artificially  made  equal  to  D>.(  See 
Flgur®  14),  The  threshold  Is  empirically  fixed  at  1/16  of 
the  total  number  of  samples  In  the  largest  square  Inscribed 
In  the  window  In  accordance  with  the  assumption  about  the 
corner  Image  relative  size, 

After  finding  two  or  more,  maxima,  the  Intensities 
which  oound  the  Intensity  sor®*d  In  each  region  are  computed 
by  taking  the  mld-Polnts  between  the  maxima,  Intensities  -1 
and  16  are  added  to  the  | I s t  of  bounds,  There  Is  an 
option,  determined  by  a  parameter  given  to  FINDIMAG,  to 
merge  all  the  regions  but  the  one  with  highest  (or  lowest) 
intensity,  Into  one  region,  This  option  can  save  time  when 
It  Is  known  that  the  Inside  Intensity  of  the  corner  has  the 
highest  (or  lowest)  Intensity  in  the  window,  In  certain 
oases  this  serves  also  as  an  "ant  I -shadow"  device.  (See 
F I gure  15) , 

There  are  two  exit  conditions  from  SLICE  which  will 
cause  FINDlMAG  to  terminate,  The  first  occurs  when  It  Is 
discovered  that  SETCLIP  set  the  clips  at  the  same  value, 
which  means  that  the  Intensity  Is  almost  uniform  tpver  the 
window,  There  Is  no  point  computing  the  histogram  in  this 
case  since  the  noise  will  make  It  unreliable,  The  other 
condition  occurs  when  It  Is  discovered  that  the  histogram 
has  0nly  one  maximum,  N0te  that  In  this  case  the  soread  0f 
Intensities  is  larger  than  In  the  previous  case  which  means 
that  there  might  be  actually  two  reylons  In  the  window, 
(See  cases  C  &  D  In  Figure  85,  The  reason  for  losing  one  of 
the  maxima  In  cases  C  &  D  Is  that  one  or  both  regions  have 
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CLIPS  OPEN 


CUPS  CLOSED 


CLIPS 


i  — before^ 

P  p-AFTER-*|  j 


CLIPS  OPEN 


CASE  B 


Fig.  12. 

OPERATION  OF  SETCLIP 
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2 

3 

If 


3  I 

t  * 


(1)  CLIPS  OPEN 

(2)  CLIPS  CLOSED 

(3)  CLIPS  SNIPPING 


THRESHOLD  FOR  SNIPPING 


(a)  Histogram  with  open  clips. 

(b)  Histogram  with  closed  clips. 

(c)  Histogram  with  snipping  clips  . 

Fig.  lj. 

EXAMPLE  OF  THE  EFFECT  OF  SNIPPING  CLIPS 
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H(I) 


□  / 

OBJECT-* 

□ 


SHADOW 

□  >  □  /  □  ALL  BOUNDS 

BACKGROUND 

□  □  BOUNDS  SELECTED  FOP  DARK  OBJECT 


OBJECT 


;  '  A 

SHADOW 

BACKGROUND 

IMAGE  IN  WINOOW 

LINE  DRAWING 

Fig.  15. 

EXAMPLE  CF  BOUNDS  SELECTION  IN  SLICE  WHICH  IGNORES  THE  SHADOW 
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wld#r  spread  of  Intensities  than  In  casts  A  4  B,  which 
violate  one  of  the  basic  assumptions, 

Jf  It  Is  decided  to  try  harder  than  the  current 
version  of  the  corner-finder#  we  can  go  In  either  or  both  of 
the  following  waysl  (a)  Make  a  more  detailed  analysis  of  tho 
histogram,  (b)  Try  again  with  smaller  windows  and  check  if 
their  hlstoOrana  peak  at  two  different  Intensities#  To 
compare  Intensities  of  two  different  windows  the  analog 
video  signal  values  have  to  be  computed  from  the  digitized 
video  signal  and  the  c|lp  setting  In  each  case, 

STARTCORNER 

The  corner  of  the  window  wh*re  pERlMETER  starts 
checking  for  boundary  oolnts  Is  selected,  It  wl||  be  easier 
to  descrloe  this  blook  later, 

PERIMETER 

Using  the  assumptions  that  the  sides  of  the  corner 
Intersect  the  window  perimeter  .hd  that  the  sides  are 
actually  the  boundaries  between  regions#  the  points  along 
the  perimeter  are  checked  In  CCW  order  starting  a*  a  corner 
of  the  window  chosen  In  STARTCORNER,  Using  the  bounds 
oomputed  |n  SLICE#  the  region  which  the  starting  point 
belongs  to  Is  established  and  oonsecutlv*  points  along  the 
perimeter  are  checked  to  see  If  they  belong  to  the  same 
region,  If  the  next  point  belongs  to  a  different  region  the 
block  FOU.OW  Is  entered  and  th*  boundary  Is  followed  until 
It  Intersects  the  perimeter  again. 

After  the  boundary  Information  Is  Prooessed  and 
there  Is  a  need  to  find  another  boundary#  either  because 
this  boundary  was  rejected  or  because  FINDIMAG  was  told  to 
find  more  than  one  feature  Inside  the  window,  PERIMETER  Is 
entered  again.  The  starting  point  now  Is  the  oolnt  next  to 
the  one  where  the  perimeter  was  left  last  to  enter  FOLLOW, 
(See  F I gure  16(a) ) , 

In  order  not  to  trace  each  boundary  twice,  once 
coming  from  on#  refllon  and  then  again  coming  from  the  other 
region,  FOLLOW  Is  entered  only  If  the  next  point  belongs  to 
a  region  of  higher  Intensity  In  the  case  that  we  are  looking 
for  a  corner  with  low  Inside  Intensity#  or  by  convention 
when  using  a||  the  bounds,  and  v|ce-versa  when  we  are 
looking  for  high  Inside  Intensity,  These  particular 
directions  ars  picked  because  It  Is  assumed  that  the 
background  Is  more  "noisy"  than  the  object's  actual 
Intensity#  and  hence  we  wl||  get  a  smoother  boundary  by 
traelng  It  from  the  Inside,  When  all  tounds  are  used,  the 
convention  mentioned  above  w|||  in  some  cases  cause  the  loss 
of  a  part  of  a  oomplex  oorner,(See  Flgur®  16(b)),  This  part 
can  be  recovered,  however,  by  analyzing  the  same  window 
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STARTING  CORNER 


v"--y 


“  CELLS  CHECKED  IN  PERIMETER 

BOUNDARY  POINTS  FOUND 
IN  FOLLOW 

— - ENTERING  PERIMETER  AGAIN 


I 


ilftl;: . J, 


_LLLOIkJij.II 


STARTING  CORNER 


UNDETECTED  EDGE 


(a)  Example  of  the  order  fn  which  cells  ire  checked  cod  boundary  points 

arc  found  (all  bounds  are  used).  *  P 

(b)  Example  of  an  undetected  edge  (all  bounds  are  used). 

Fig.  16. 

OPERATION  OF  PERIMETER 
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•Q«ln#  this  time  looking  fa*  a  corner  with  high  intensity. 
FOLLOW 

To  find  the  boundary  Dolnts  FOLLOW  uses  a 
maze-so 1  v  I  ns  trUkl  walking  wltn  the  right  hand  continuously 
touohlng  th«  wall.  Soma  addad  trlckaPy  Is  uead  to  minimize 
tha  numbar  of  oolnts  chaokad  In  ordar  to  find  the  next  DOlnt 
o<  tha  boundary  as  axp|alned  ba|ow,  No  point  Is  chacked 
twloa  If  th®r®  ar®  no  loops  or  vary  sharp  cornars, 

Aft«r  a  point  on  tha  boundary  Is  found#  Its  8 
nalghbours  ar®  oh«ck«d  (tha  samplas  ara  spaced  on  a 
r®otangular  grid)  In  CCW  ordar,  The  first  point  which 
ba longs  In  th®  same  rag|on  as  tha  currant  on®  Is  the  naxt 
boundary  point,  The  dlraotlon  of  tha  nalghbour  point  which 
will  be  checked  first  Is  th®  diagonal  In  CW  direction  from 
tha  direction  from  the  last  boundary  Point  to  the  oresent 
point.  (See  r|gures  17(a)  and  17(b)), 

FOLLOW  will  automatically  ent®r  PERIMETER  Again  in 
the  f o 1  I  owi  ng  o®s®si 


a)  Th®  first  point  chaokad  belongs  to  a 
different  from  those  of  all  Its  8  neighbours,  Such  a 
Is  called  a  singular  point,  (See  Figure  17(c)). 


rag  on 
oo  nt 


<b)  Thf  boundary  loops  on  ltse|f  and  Intersects 

first 


.  >a  bounaafy 

perimeter  aga|n  exactly  at  Its  first  point, 
looping  Is  different  from  local  loops  whloh  are 
the  next  block,  (See  Flgur®  17(d)), 


ersects  the 
This  kind  of 


pruned 


(c)  The  number  of  oolnts  found.on  the  boundary  Is 
smaller  than  a  threshold  empirically  fixed  at  half  the 
smaller  dimension  of  the  window,  In  accordance  wjth  the 
assumption  that  the  Image  of  tb®  corner  ocoucy  more  than  a 
o®r ta 1 n  part  of  the  window  area,  Ideally  this  r®glon 
should  ba  ru|ed  out  by  tha  threshold  In  SLICE  but  thefe  are 
eas®8  whan  w®  have  In  one  region  a  small  patch  of 
Intensities  In  the  same  Interval  as  another  region, (Sea 
Figure  17(e)),  Note  that  the  above  threshold  corresponds  to 
a  square  region  whose  area  Is  the  threshold  In  SLICE,  Any 
region  with  the  same  area  whloh  "Invades"  tha  window  more 
will  have  *  longer  boundary  and  thus  would  not  be  rejected 
by  FOLLOW, 


FOLLOW  uses  two  kinds  of  data  structures  in  Its 
operation,  One  Is  a  raotangular  array  with  the  dimensions 
the  window,  When  a  point  at  window  coordinates  l»J  Is  found 
aa  a  boundary  po|nt,  a  1  |s  stored  In  eflll  l#J  In  the  array, 
s0  that  If  a  Dolnt  Is  f0und  again  a  1 0ca I  lOOD  l9  Indicated. 
Tha  other  structure  Is  <4  linear  array  In  which  the 
coordinates  of  a  boundary  point  and  tha  above  numbar  Is 
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D  I  :  DIRECTION  FROM  LAST  BOUNDARY 
POINT  TO  PRESENT  POINT 


D  2  :  DIRECTION  FROM  PRESENT  BOUNDARY 
POINT  TO  NEXT  POINT  TO  BE  CHECKED 


(b) 


— POINTS  CHECKED  (IN  ORDER  OF  CHECKING) 
- »  BOUNDARY  POINTS  FOUND 

(a)  Order  of  points  checked  in  FOLLOW. 

(b)  Examples  of  'Viazo-walking". 


Fig.  17. 

OPERATION  OF  FOLLOW 
(Continued  on  next  page) 


(c)  Singular  point. 

(d)  Loop. 

(e)  Patch. 


Fig.  17. 

OPERATION  OF  FOLLOW 
(Continued) 
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(a) 


f 
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(c) 


(a)  Original  line  and  noise. 

(b)  Boundary  points  and  fitted  line  (rejected). 

(c)  Boundary  points  and  fitted  corner  (also  rejected 


Fig.  19. 

EXAMPLE  OF  A  VERTEX  OUTSIDE  THE  WINDOW 


definitions*#  direction  cosines  of  the  second  side#  location 
of  the  vertex  (In  oase  of  a  line  it  Is  the  midpoint)#  aj  I  in 
Imago  coordinate  system#  and  whether  a  line  or  a  comer  was 
found  <1  or  0  accordingly)# 

A,  fine  Is  tried  before  a  come;-  beoaute  if  the 
boundary  Is  really  a  line#  a  corner  would  fit  t  at  least  as 
we||  as  a  Mne  <l,e  with  smaller  average  erm<  .  The  vertex 
in  this  oase  would  be  In  an  arbitrary  looa-Jon  along  the 
line,  However  the  Information  that  the  border  Is  indeed  a 
line  Is  Important  and  should  be  Preserved,  Alternately  a 
corner  could  be  fitted  first  and  then  a  line  should  be  tried 
only  If  the  Inside  angle  of  the  corner  Is  oiose  to  *  or  one 
of  the  sides  Is  muoh  shorter  then  the  other, 

We  mentioned  already  that  a  patch  on  the  perimeter 
will  be  rejected  beqauee  Its  outline  Is  not  long  enough,  If 
the  patoh  Is  on  the  boundary  It  will  either  be  Pruned  out  as 
a  branch  or  a  loop#  or  It  will  slightly  perturb  the  line 
fitted  to  the  boundary  points#  A  oatch  In  the  middle  of  a 
region  would  not  be  seen  at  all,  although  It  oould  affect 
the  Intensity  histogram, 


The  situations  that  can  cause  failure  although  two 
peaks  have  been  found  In  the  histogram  are:  (a)  "Salt  and 
pepper"  texture,  <b)  A  boundary  oontalnlng  more  than  one 
oorner  Inside  the  window,  Note  that  both  the?e  cases 
contradict  the  basic  assumptions  so  that  failure  is  not 
surer  Islng, 


RELINT 

If  the  boundary  points  were  fitted  by  a  corner  in 
CORNFIT  and  If  a  relative  Intensity  matoh  Is  demanded, It  is 
done  In  this  block,  To  find  the  relative  Inside  intensity, 
RELINT  uses  the  following  fact:  If  the  two  highest  bounds 
are  used,  and  If  the  first  side  was  fitted  by  the  boundary 
points  found  first  In  FOLLOW,  then  the  Inside  Intensity  is 
high,  This  situation  Is  oaused  by  the  definition  of  corner 
sides  and  by  the  conventions  established  In  PERIMETER, 
Similar  considerations  are  made  for  the  other  combinations 
of  bounds  used  and  Inside  relative  intensity,  (See  Figures 
16  and  17  for  examples), 

If  the  relative  Intensity  Is  ohecked  and  a  match  is 
not  aohleved,  PERIMETER  (s  entered  again, 

RECENTER 

If  the  location  of  the  corner  or  the  midpoint  of  a 
line  Is  not  oiose  enough  to  the  center  of. the  window  (the 
threshold  Is  empirically  set  at  1/8  of  the  dimension  of  the 
window  In  each  direction),  and  if  recenterlng  Is  permitted 
(as  determined  by  one  of  the  parameters),  the  window  is 
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wovtd  30  that  Its  osntir  w|||  coincide  with  the  vertex  (or 
(rldpolnt)  and  the  oomolete  Process  Is  started  again,  This  Is 
done  partly  because  If  the  comer  Is  not  centered,  one  of 
the  sides  may  be  short  and  the  line  fitted  to  It  inaccurate, 
Other  reasons  will  beoome  aoparent  In  the  discussion  of 
SRCHIMAG, 

COLT 

In  this  block  the  form  of  the  feature  found  is 
iratohed  If  a  match  Is  demanded,  The  form  of  the  feature  to 
match  aOalmst  Is  given  In  the  same  data  structure  that  Is 
used  In  CORNFIT,  A  matoh  Is  achieved  If  the  oosine  of  the 
angle  between  the  two  sides  to  be  matohed  Is  closer  to  l 
than  a  threshold  given  by  a  parameter.  The  oosine  of  the 
angle  Is  chosen  as  a  criterion  since  It  Is  easily  comouteo 
from  the  direction  ooslnes  stored  In  the  data  structure  and 
Is  Invariant  to  angle  direction, 

Three  Kinds  of  matches  can  be  demanded,  determined 
by  a  number  storad  In  tha  3th  cell  of  the  given  feature  data 
struoturei 

(a)  Both  sldas  have  to  be  matched, 

<b)  At  least  one  of  the  found  sides  has  to  matih  the 
first  (second)  side  of  the  given  feature, 

(c)  At  least  one  of  thf  found  sides  has  to  match 
either  of  the  sides  of  the  g|v*n  feature, 

If  there  Is  no  match  In  the  last  two  cases  PERIMETER 
Is  entered  again,  If  thar*  Is  no  match  In  case  (a)  a 
oontra-matoh  ]s  triad!  wa  ohack  to  see  If  any  of  the  sides 
found. has  a  direction  oboslte  fwlthln  the  above  tolerance) 
to  either  of  the  given  feature  sld&s,  This  Information  Is 
needed  to  aid  the  searoh  for  the  feature  (see  discussion  of 
SRCHIMAG),  If  no  contra-match  Is  founa,  PERIMETER  Is 
anttrtd  again. 

Tha  output  of  COLT  Is  stored  In  the  8th  cell  of  the 
linear  array  mentioned  already  In  CORNFIT  as  follows!  -1  If 
no  match  was  obtained,  3  If  a  match  was  obtained  In  (a),  or 
tha  side  that  matohad  <1  or  2)  In  (b)  and  <o),  or  the  side 
that  contra-matched  In  (a), 

startcorner 

rErImETEr  begins  to  ohack  the  points  at  a  corner  of 
the  window,  If  the  form  of  tha  oorner  sought  Is  known,  i.e, 
stored  In  the  data  struotura  of  the  oorner  to  match  against, 
wa  oan  select  a  starting  corner  which  will  minimize  the 
number  of  points  ohecked  In  PERIMETER  before  FOLLOW  ls 
entered,  In  the  following  way!  In  most  situations  we  would 
like  to  trace  the  boundary  from  tha  Inside  of  the  corner, 
because  we  assume  that  tha  background  Is  more  noisy. 
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Therefore  we  select  the  first  comer  of  the  window  In  ?. 
olookwlse  direction  from  a'|  possible  positions  of  the 
first  side  of  the  given  oorner#  es  the  stertlng  point  for 
PERIMETER,  (See  Figure  20), 


MORE  will  exit  under  any  of  the  following 
cond I t i ons : 

(a)  The  number  of  oorners  (or  lines)  that  FINOIMAG 
was  told  to  find,  have  been  found  and  properly  matched, 

(b)  A  oorner  Is  sought#  there  Is  no  demand  to  match 
form,  searching  Is  permitted  and  a  line  was  found, 

(c)  A  feature  Is  sought  while  searching  Is  permitted 
end  a  oorner  or  a  line  was  found  which  oontr a-matohed  the 
given  feature  form, 

Otherwise  PERIMETER  Is  entered  again, 

3.2,3  STRUCTURE  OF  SRCHI MAGE 

The  block  diagram  of  SRCHI MAG  Is  shown  In  Figure  21. 
The  description  of  the  blocks  follows. 


If  search  Is  not  permitted,  FJNDIMAG  Is  entered  and 
then  SEARCH  exits,  If  search  Is  permitted,  It  Is  done  In  a 
spiral  scan  pattern.  The  first  position  of  the  analyzed 
window  Is  chosen  to  b«  a*  the  most  Provable  olao«  to  find 
the  featur®  as  determined  by  Information  which  SEARCH  has 
•ocsss  to,  From  this  position  ths  seaPoh  spirals  outwards 
with  overlapping,  (see  Figure  22),  At  eaoh  position  FINOIMAG 
Is  ealled,  The  spiral  search  Is  continued  Until  on*  of  the 
following  conditions  Is  satisfied! 

(a)  The  oorner  or  line  searched  for  Is  found  and 
properly  matched,  In  this  oa3e  SEARCH  exits, 

(b)  A  feature  Is  searched  for  with  form  match  and  a 
corner  or  a  line  Is  found  wh!c-h  contra-matched  the  given 
feature,  In  this  case  FOLMATCH  is  entered, 

<c)  A  corner  In  searched  for,  no  form  metoh  Is 
needed  and  a  line  was  found,  In  this  case  FOLINt  Is  entered, 

(d)  The  next  window  position  Is  partly  outside  the 
permitted  search  space,  In  this  case  SEARCH  exits  and  a 
failure  Is  Indicated,  Ths  search  spaos  Is  a  rectangular  part 
of  the  frame,  Its  oenter  coincides  with  the  first  window 
center  and  Its  size  Is  determined  by  a  parameter  given  to 
the  module, 


FOLINE 
stored , 


The  position  of  the  window  when  FOLINE  Is  entered  is 
Then  the  window  oenter  Is  movtd  along  the  direotlon 
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EXIT  EXIT 


Fig.  21. 

BLOCK-DIAGRAM  OF  SRCHIMAG 


of  the  lino  by  steps  of  ,5*<  window  s l z •  ),  If  a  line  Is 
found  again,  the  window  continues  to  move  alonfl  the  line 
till  It  r«aohes  the  eearoh  eoaoe  border.  Then  the  line  is 
followed  In  the  other  direction  from  the  Doeltlon  stored 
wher  FOUNE  was  first  entered  till  the  search  space  border 
on  the  other  side  Is  reached,  In  this  case  FOUNT  exits  and 
failure  Is  I nd j eated • 

If  a  corner  which  matohes  Properly  <not  a  form 
match)  Is  found.  FOIINE  exits  successfully.  If  at  sny 
position  nothlnd  Is  found  In  the  analyzed  window,  or  p. 
corner  was  found  which  does  not  match  properly,  the  search 
Is  abandoned  In  this  dlreotlon,  If  It  was  the  first 
direction  tried,  the  opposite  direction  Is  searched  as 
above,  Otherwise  FOUNi,'  exits  and  failure  Is  Indicated, 


Th#  window  center  Is  moved  alono  the  direction  of 
the  side  that  contra"*',atohed .  by  ,5*(  window  size  )  if  tne 
side  belongs  to  a  line  or  by  <2/3)*<wlndow  size)  if  It 
be | onfls  to  a  corner,  This  difference  assures  that  In  the 
next  window  position  the  corner  would  not  be  seen  at  all, 
otherwise  the  window  would  Be  recenter®d  on  the  corner  and 
FOLMATCH  would  loop  miserably,  If  another  feature  with 
contra-matching  side  Is  found  In  the  new  window  position,  It 
Is  followed  as  above,  Otherwise  FOLMATCH  exits,  successfully 
If  a  feature  was  found  and  matohed  properly,  with  failure 
Indication  If  not,  <See  Figure  23), 

Figure  24  gives  an  example  whloh  shows  why 
recenterlng  Is  so  Important  for  searching  when  form  matoh  is 
needod,  although  It  can  cause  looping  If  proper  care  Is  not 
taken. 

3,3  User  parameters  and  oorner-f  I nder  module  output 

There  are  two  kinds  of  parameters  which  the. user  has 
to  supply  to  the  corner  finder,  global  and  formal,  The 
gl0ba|  parameters  are* 

(a)  The  number  of  the  oamera  to  be  used,  Currently 
only  one  oamera  Is  used  but  shortly  there  will  be  two. 

(b)  Window  ep«o I f 1  oat  I ons i  the  coordinates  of  the 
center  In  the  l,c,s,»  width  and  height, 

These  parameters  occupy  In  that  order  the  first  5  of 
an  8  cell  linear  array  (oalled  LOOK.AT),  The  other  3 
contain  the  current  setting  of  the  clips  (BCUP  and  TCUP) 
and  the  target  voltage,  These  8  parameters  are  needed  (  In 
pr I  no  I  p I e)  to  oomoute  the  light  Intenelty  distribution  on 
the  portion  of  the  vldlcon  face,  corresponding  to  the 
window,  from  the  Intensities  stored  at  the  sampling  points, 
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(o) 

Window  size 


(b) 

Given  corner  form 


Sequence  of  window  positions 

(1)  Corner  (wrong  one)  found. 

(?)  Rccentering. 

(j5)  Side  ?  contra-matched  and  followed. 

(4)  Line  followed. 

(5)  Corner  found. 

(6)  Recentering. 


Fig.  2^. 

EXAMPLE  OF  CONTRA-MATCH  FOLLOWING 
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(1)  Original  window  and  fitted  line. 


(2)  Contra-match  followed  without  rccentering, 
points  found. 


Not  enough  boundary 


(3)  Window  rccentered  and  the  corner  found. 


Fig.  2U. 

EXAMPLE  OF  TUE  NEED  TO  RECENTER 


For  those  who  will  aotu&lly  us®  the  corner-f I nd#r 
programs  I  w|  |  |  us®  thr  namas  of  the  formal  parameters  a$ 
they  appear  In  th®  progrcm,  The  names  do  not  necessarily 
express  their  meaning,  They  arei 

(a)  BKGR  (see  dlsscusslon  of  SLICE  In  Section 
3,2,2).  BKGq  can  have  three  values;  -1  when  we  want  to  us* 
the  two  highest  pounds*  for  example  In  the  case  that  we  know 
that  the  corner  Is  relatively  light;  1  when  we  want  to  use 
the  two  lowest,  bounds*  for  example  In  the  case  that  we  know 
that  the  corner  Is  relatively  dark;  3  when  we  want  to  use 
all  the  bounds*  for  example  In  the  case  that  the  rotative 
Intensity  Is  not  known  or  that  we  are  looking  for  more  than 
one  feature* 

(b)  I  NT  (see  discussion  of  RELINT  In  Section  1,2,2). 
j NT  can  have  three  values*  -1  when  we  want  to  match 
relative  Intensity  and  we  are  looking  for  lighter  Inside;  l 
when  we  want  to  match  relative  Intensity  and  we  are  looklno 
for  darker  Inside;  when  we  do  not  demand  a  match  of 
relative  Intensity,  BKGR  and  I  NT  do  not  necessarily  have 
the  same  value  although  they  usually  do.  For  example  if  wo 
want  to  trace  a  dark  corner  from  the  outside  but  natch  Its 
relative  Intensity*  and  It  Is  expected  that  we  will  have 
only  two  regions  In  the  window,  then  we  w| | |  set  BKGR  to  -1. 
but  INT  to  1, 


(c)  SEARCH  (See  discussion  of  RECENTER  end  MORE  In 
Section  3,2,2  and  of  the  block  SEARCH  In  Section  3,2.3),  If 
SEARCH>0,  r®c«nterlng  end  searching  are  permitted.  Tne 
olmenslons  of  the  setrch  spac®  are  then  given  by  the  vslue 
of  SEARCH+i  times  the  dimensions  of  the  window  used,  The 
number  of  windows  needed  by  the  spiral  scan  to  cover  tne 
whole  search  space  Isi  ( 2«SE ARCH+J, )  *2,  If  SEARCHs^, 


recenter  I ng 


_ _ _  _  permitted  but  not  searohlng,  If  sE ARCH<2 

neither  searching  nor  r®centerlng  Is  permitted.  In  this  case 
-SEARCH  Is  the  number  of  features  (corners  and/cr  lines) 
wanted  Inthe  window,  Implicitly  only  one  feature  is 

sought  If  SEARCHES,  (Se#  discussion  of  slmole  and 

cluttered  environments  In  Section  3*1,3), 


(d)  TOLER  (see  discussion  of  COLT  In  section  3,2.2), 
If  TOLERO  no  match  of  form  Is  demanded,  If  T0LER>2,  -atch 
0f  f0rm  Is  demanded  and  the  value  of  TOLER  Is  the  tcl«rance. 
Since  the  to|erance  Is  compared  with  the  difference  between 
1  and  the  cosine  of  the  angle  between  the  two  sides  to  oe 
matched*  values  of  TOLER  larger  than  1  are  equivalent  to  no 
damand  for  match,  Tha  form  of  the  feature  to  natch  against 
Is  given  In  the  next  and  last  parameter. 


(  e )  0 1 RC  Is  an  8  oe||  | | n  e  a  r  array, 
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"he  first  4 


calls  conta I r  th#  dlreotlon  ooslnss  of  ths  two  sides  of  the 
given  faatum,  The  next  2  contain  Dredloted  coordinates  of 
the  vertex  of  the  oorner,  They  »r#  not  uMd  In  the  match 
JrSoIi;,  Th.  7th  cell  dt.rmln.s  If  a  line  (1)  or  a  corner 
(0)  Is  sought.  The  3th  ceil  determines  whether  we  demand  e 
rr.tch  of  both  eldea  <0>»  flret  side  orny  (Hi  second  side 
l*near  arrays*  on*  for  each  feature  If  SEARCH<0,  described 

When  the  comer-finder  Is  working  under  the  hand-eye 
monitor  It  Is  aotlvated  by  the  follwlng  message  procedure: 

SRCH-1MAG<BKGR, ! NT, SEARCH, TOLER, D I R0> I  .  , 

? n  this  case  the  following  outputs  reside  In  olobal  cells. 

The  output  of  the  corner-finder  Is  a  set  of  8  cell 
linear  arrays,  one  for  *aoh  featur*  if  SEARCH<0,  descr  bep 
In  C0RNF1T  and  COLT  In  Section  3.2.2.  in  MORE  It  Is  dec  ded 
If  e  feeture,  which  h  stored  In  the  last  array  » 

satisfies  a  I  I  the  demands,  otherwise  this  array  Is  erased 
from  the  aet.  If  the  corner-f Inder  failed,  the  reason  for 
failure  Is  stored  Us  *  number)  In  a  special  cell  (cal  ed 
EYEFLG),  For  debugging  purposes  many  Intermediate  results 
are  displayed  during  Program  execution,  (See  Figure  25  or 
an  axamp I e ) , 


a 


O 


j 
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Fig.  25. 

CORNER-FINDER  MODULE  RESULTS  DISPLAY 


CHAPTER  41  calibration  updating 

4.1  Introduction 

4.1.1  General  notes 

The  purpose  of  the  calibration  updating  programs  is 
to  maintain  the  acouracy  of  the  camera  model  against  changes 

°V#f  we  start  the  discussion  by  describing  in  detail  the 
general  causes  of  the  Inacouraeles  of  the  hand-eye  system, 
especially  those  affecting  the  precision  of  manipulations. 
Then  we  descrifce  the  oamera  model  and  Its  calibration 
programs  as  developed  by  I,  Sobei  C263,  Analysis  of  the 
causes  of  variations  leads  to  the  description  of  the 
updating  of  the  pan  tilt  and  focus  parameters. 


4,1,2  Hand  eye  coordination 

Let  us  define  a  nominal  coordinate  system  <o,s.) 
which  we  call  the  ’’table  coordinate  system"  (t.c.s,),  (see 
Figures  1  and  2 6)»  as  an  orthogonal  right-handed  three 
dimensional  system  with* 

* 

(a)  The  plane  z*0  Is  ’’close"  to  the  face  of  the 

table,  . 

(b)  The  origin  is  ’’close"  to  the  corner  of  the 

table  near  the  arm,  ^  u 

(c)  The  x  and  y  axes  ar»  "close’’  to  the  edges  of  the 
table,  (x  going  upwards  In  Figure  1  and  y  to  the  left), 


Each  point 
trlol*  of  numb< 


jomx  In  space  can  be  unlpue|y  defined  by  a 
numbers  which  are  Its  coordinates  in  she  t.c.s.  We 
(the  users)  do  not  have  any  means  of  finding  this  triple  of 
numbers  since  the  t.o.s.  Is  hypothetical.  We  can  get 

apBroxlm»tlons  by  the  u9e  pf  s0me  assumptl0ns  <fer  examp  e, 
the  planarity  of  the  table's  faoe)  and  some  measuring 
devices  like  rulers,  e  right  angle  trlood  and  a  grid  paper 
glusd  on  the  table's  face,  We  will  oall  this  approximation 
the  ’’user-table  coordinates"  (u,t,c»)  of  the  point, 


The  hend-eye  system  has  two  ways  to  find 
approximations  to  the  coordinates  of  a  point  in  the  t,o,s,t 
One  Is  by  using  the  camera(s)  and  tha  other  by  using  the 
arm, 

First  we  will  discuss  the  determination  of  3-D 
coordinates  using  the  camtra(s)i  The  use  of  one  view*  from 
one  camera*  Is  Inharently  not  enough  to  determine  3-D 
coordinates,  This  problem  I?  discussed  extensively 
elsewhere  C’6],  I  would  like  however  to  comment  on  four 
possible  solutions! 
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(a)  Stereo-  I .  •  •  the  use  of  two  view*  obtained  with 
on#  or  two  cameras  from  two  Points  which  #r#  net  collnear 
with  the  oolnt  to  be  measured.  Th#  main  problems  with  this 
solution  are  acouraoy  for  "narrow  angle"  stereo  and 
ce-’-sspondnr-e  for  "wM*  ang»e"  sterer  C2tJ, 

(b)  Focus  Information-  The  automatic  focussing 
module  can  be  used  here.  Its  acouraoy  can  be  estimated  py 
the  minimal  deoth  of  field  Information  given  In  Appendix  B, 
(.5  to  1  Inch)  which  Is  not  enough,  see  also  U13. 


(c)  S.tad  lametr  I  o  Information-  1,8,  the  use  of  Known 
dimensions  of  objeots,  This  method  Is  oomparable  In  accuracy 
to  that  mentioned  above  for  the  automatic  foousslng  program. 
The  acouraoy  here  Is  limited  by  the  accuracy  of  the  scene 
analyzer  and  Inherently  oy  the  resolution  of  the  sensor, 


(d)  Support  hypothesis-  l,e.  supplying  external 
Information  that  the  point  lies  on  a  partlouiar  olane  or 
line  spoe I f fed  In  the  t.c.s.  This  method  Is  the  one  used 
extensively  In  the  hand-eye  system  In  ways  described  In  this 
report.  (see  also  C22]  and  C 8 3 > • 


A  fifth,  somewhat  unrelated,  method  uses  controlled 
Illumination  of  the  soene,  This  method  was  Implemented  oy 
Aflln  at  Stanford,  using  a  planar  laser  beam  to  scan  the 
soeneCl? , 


The  most  oommon  application  of  the  support 
hypothesis  Is  when  a  point  Has  on  the  surface  of  the  table 
which  wa  assume  Is  the  plane  Zs0,  The  aporoxlmat  on  computed 
by  the  eamera  using  the  support  hypothesis  we  w III  call  the 
"oamera-tab I e  coordinates"  (c.t.c,)  of  a  point,  The 
difference  between  the  coordinates  of  a  point  in  the  t.c.s. 
(which  we  ©an  not  measurt)  and  Its  c,t,c,  have  the  following 
oaustii 


(a)  tha  quality  of  the  assumption  that  the  table 
surface  Is  the  plane  z*0, 

(b)  the  quality  of  the  Image, 

(c)  the  precision  of  the  soene  analyzing  routine, 

(d)  the  Quality  of  the  mathematical  model  of  the 


camera, 
mod*  | , 


(e)  the  precision  of  the  Invariant  parameters  of  the 

(f)  the  quality  of  the  mesurement  of  the  variable 


parameters  of  the  model, 


(a) , (e ) , ( d)  and  (a)  contribute  tha  oetermlnlstlo 
component  of  tha  difference.  (b)  and  If)  contribute  the 
random  part  of  tha  d|ff#ranoe,  Whan  we  approximate 
differences  of  coordinates  of  two  olose  points,  using  their 
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Images  In  the  same  frame,  the  main  contributions  to  the 
error  are  (b>,  (c)  and  parts  of  (d)  and  (e),  The  purpose  of 
the  camera  calibration  Program  developed  by  I.  Sobel  is  to 
reduce  this  difference  by  Improving  the  precision  of  the 
Invariant  parameters  of  the  model,  Some  of  the  parameters 
assumed  Invariant  are  varying  over  time,  One  of  the  purposes 
of  the  calibration  updating  progams  Is  to  "track"  these 
variations,  The  remaining  variations  add  to  the  random 
component  of  the  dJf?erenoe, 

We  now  proceed  to  describe  the  determination  of  3-D 
coordinates  using  the  arm,  When  the  arm  control  program  is 
given  a  command  to  place  the  arm  at  some  position,  It  tries 
to  DUt  the  midpoint  between  the  tips  of  the  touch  sensors  at 
this  position.  Using  the  arm  model  and  the  reading  of  the 
potentiometers  on  the  arm  Joints,  we  can  compute  an 
approximation  for  the  coordinates  in  the  t.c.s.  of  any  point 
whose  position  relative  to  the  hand  is  known,  For  example, 
If  we  assume  that  we  know  the  exact  opening  between  the 
finders  and  the  thickness  of  the  rubber  pad  of  the  touch 
sensors,  among  other  dimensions*  we  can  compute  the  location 
of  the  hand-mark,  We  w|||  call  this  approximation 
"hand-tabis  coordinates'*  <h,t.0»). 

Note  that  when  a  cube  of  known  dimensions  Is  grasped 
by  the  hand,  we  know  the  relation  of  Its  corners  to  the 
reference  point  of  the  arm  only  In  the  direction  of  the 
sweeping  motion  of  the  fingers  (perpendicular  to  the  finger 
face)  but  not  In  the  other  tuo  directions,  Hence  the  arm  can 
not  be  used  as  a  general  measuring  device,  and  we  use  the 
approximation  given  by  the  c,t,c,  as  our  standard, 

Tha  causes  of  the  differences  between  the 
coordinates  of  a  point  in  the  t,c,s.  In  Its  h,t»c,  are 
similar  to  oaus«s  (d)  through  (f)  given  for  the  camera,  The 
arm  calibration  program  tries  to  reduce  this  difference  by 
Improving  the  values  of  the  Invariant  parameters  of  the  arm 
model , 


The  differences  between  the  c.t.c,  of  a  point  and 
Its  h.t.o,  have  muoh  more  Importance  to  us  than  the 
differences  between  either  of  them  and  the  coordinates  of  a 
point  In  the  nominal  t,c.s,  (or  its  u,t,o,)*  since  the 
difference  between  the  c,t*c.  and  the  h.t.c,  directly 
affect  the  Precision  of  manipulation.  We  will  call  this 
difference,  "coordination  error",  One  of  the  purposes 
of  the  calibration  updating  Programs  is  to  reduce  the 
coordination  errors,  In  the  visual  feedback  tasks  the 
residual  coordination  error  Is  measured  In  each  situation 
and  then  reduced  (below  a  given  threshold)  by  Incrementally 
specified  movements  of  the  arm, 
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From  now  on#  when  *•  mention  coordinates  In  the 
t.c#9.»  we  actually  mean  an  approximation#  Hopefully  It 
Mill  be  clear  from  the  context  which  approximation  we  mean 
( u ,  t « c « #  c 1 1 1 c i  or  h. t# c, ) . 


4,2  Camera  model  and  calibration 
4(?,1  E«m*fa  'iqcs*U  g*n*r * '  net*i 

The  camera  moa*|  has  1*0  parti  which  ire  oil  lip  the 
a*t*rn«!  and  the  Internal  model*  The  external  model  deals 
ultn  the  Parameters  which  affect  the  transformation  of  the 
external  span*  to  an  Intig*  formed  on  the  video*  face*  The 
I  n  m  rna  l  model  dtagi-lbes  the  scanning  and  sampling  nrocesees 
p*  -hie*  trie  Into#  I*  converted  to  readings  at  t^e  sampllnp 
points.  me  model  fa  geometric  lh  nature  and  "Ot 
photometric*  the  model  *|iow3  us  to  compute  for  sic'-  aolr.t 
In  t-n*  external  world#  specified  for  exemoli  by  Ito  3 
coordinate!  In  tne  "tibia  coordinate  s>itemh  It.p,s,)i  the 
tilling  point  In  tne  IrC-i,  wnkh  correeoondi  to  itch  point 
arid  v*v,  Mn  thli  cea#  It  «Ul  o*  a  f*y  <n  tha  t.c,s<  ano 
nat  a  po 1 "t* ♦  . 

The  sionetrla  model  ana  Its  celloretlon  pracidir* 
wer*  devtioced  by  I,  loo*  I  CZO),  *  photometric  "poo'  wip 
developed  &y  Jt  H*  Tenenbegm  tZfl]. 

4,2*2  Cah»r*  external  model 

The  pan/tllt  head  and  focussing  mechanism  are 
de$orlbed  In  the  model  as  follows  (see  Figure  26): 

(a)  The  pan  axis  Is  perpendicular  to  the  "table  top" 
(this  is  the  name  given  to  the  Plane  Zs0  In  the  t,c,s,)«  end 
pleroes  It  at  point  (P1#P2,0), 

(b)  The  tilt  exls  Is  perpendicular  to  the  can  axis 
and  Intersects  It  at  point  (P.1#P2»P3), 

(o)  The  looetlcn  of  the  lens  center  and  the 
direction  of  the  principal  ray  Is  described  as  follows:  we 
define  a  ooordlnate  system  which  will  be  called  the  "camera 
coordinate  system"  (o,c«s«)  so  that  Its  origin  Is  at  the 
lens  center,  Its  rotation  relative  to  the  t.c.s,  Is  defined 
by  the  pan  and  tilt  angles  and  a  fixed  rotation  angle  around 
the  principal  ray  called  SWING,  The  z  axis  Is  in  the 
direction  of  the  principal  ray,  the  y  axis  Is  In  a  Diane 
almost  (because  of  SWING  whloh  Is  small)  perpendicular  to 
the  tab!*  too  and  pointing  towards  the  table  top,  The  x  axis 
completes  the  c.s,  to  be  a  r,h,3,  The  Intersection  of  the 
pan  and  tilt  axes-Us  located  at  Dolnt  (-0P1#-DP?,-DP3)  In 
the  o.c.s.  Note  that  focussing  Is  done  by  moving  the  vidlcon 
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Fig.  26. 

CAMERA  EXTERNAL  MODEL 
(Drawn  for  SWING  =  0) 


\ 
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and  thus  DPI , DP2, DP3  remain  fixed  fop  a  given  lens, 

Idl  Ths  yldMon  fast  fi  cir  pend  1  cu  I  *  r  li  Jht 
or  Inc i &•  I  **y  # n£J  I*  I  sea  tad  at  a  distinct,  t'i'en  f  Or 
oen  enlence  «l|i  b*  tailed  "foe**  fra*  true  lent  cantor  in 
th#  t*0lpiet|on  of  th«  It  Is  assumed  that  thi  astlri 
can  b*  utiicflotd  by  cn*  (Imol*  itngi*  Ians  Sac^ttrlc  mod*) 
*Uh  tha  fndal  lanBth  toaoMl*C  on  tha  |en 9  assembly, 

fa;  in*  sin  ana  tilt  ■  n 0 1  •  *  tnd  t+*  dstarci  foe  can 
not  be  read  d|r»stly  ay  tha  computer .  [ngtiiJ,  three 
potentiometers  (pots)  are  usad,  two  0  I r  0  u | a  r  and  one  linear, 
They  ar#  connected  to  a  regulated  power  supply,  The 
voltages  at  thslr  wipers  are  converted  to  digital  form  and 
stored  when  neccesary,  We  assume  that  the  pot  readings  are 
linear  with  the  pan  and  tilt  andjas  and  foo,  and  hence  6 
parameters,  scale  and  offset  for  each,  are  needed  to  convert 
the  readings  to  pan  tilt  and  foe,  These  parameters  are 
called  PPOT0,  PPOTD,  TPOT0,  TPQTD,  FQCLEN0,  FOCIENG. 


The  camera  external  model  Is 
parameters:  Pi,  P2,  P3,  SWING,  DPI, 
TPOT0,  TPOTD ,  TOCLEN0,  FONLENg, 


thus  specified 
DP2,  DP3,  PPCT0, 


4.2,3  Camera  Internal  mode) 

The  scanning  and  sampling  prooasses  are  desorlbedin 
the  mode)  as  follows  (sea  Figure  27): 

(a)  The  x  and  y  axes  of  the  l.c.s.  are  parallel  to 
the  x,y  axes  of  the  c.c.s. 

(b)  The  principal  ray  pierces  the  Image  plane  at 
point  (PP1.PP2)  of  the  l.o.s, 

(c)  The  sampling  resolution  is  KX  semples/inch  In 
the  x  direction  of  the  l,c.s,  (l,e.  sampling  along  a  soan 
line)  and  ky  ,samp  I  es/ 1  nch  In  the  y  direction  of  the  i.e.s, 
(l.e  the  number  of  lines  per  Inch  scanned).  Usually  kx 
dlffepg  from  KY  by  only  a  few  percent. 

(d)  The  scanning  starts  at  the  origin  of  the  i,c«s. 
The  scan  lines  are  almost  parallel  to  the  x-axls  of  the 
l.c.8.1  since  each  line  ends  where  the  next  line  starts, 
Hence  the  |ln«s  are  at  an  angle  of  1/333  rads,  to  the 
x-axls,  Subseauent  lines  lie  In  the  y-dlrection  of  the 
l.c.s. 

(e)  The  effect  of  Image  Inversion  by  the  camera  is 
cancelled  by  the  fact  that  In  reality  the  scan  starts  at  the 
lower  right  hand  corner  of  the  frame  and  not  at  the  i.e.s, 
origin, 

The  camera  Internal  model  Is  thus  specified  by  4 
parameters:  PPi,PP2,KX,KY. 
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The  computation  of  the  transformations  between  the 
l.c.s.  and  the  t,c,3.  from  th#  17  parameters,  the  3  Dot 
readings  and  the  lens  fooal  length  Is  #xp|alned  in  detail  In 
C 26 3  and  Is  summarized  In  Appendix  A, 

Like  the  camera,  the  arm  also  has  a  mathematical 
model,  This  model  has  32  parameters  whloh  are  used,  with  the 
readings  of  pots  In  the  arm  Joints,  to  comoute  the 
transformations  from  the  t,c,s,  to  the  "hand  coordinate 
system"  (h,c.s,,  a  c,s.  tied  to  the  hand)  and  vice-versa, 

4.2,4  Camera  model  calibration  process 

There  Is  no  easy  way  to  generate  a.  oallbrated  image 
on  the  vldloon  face  without  using  an  external  object  (like  a 
T , v  ohart)  and  the  camera  optics,  Also  there  Is  no  easy  way 
to  measure  an  Image  formed  by  the  camera  optics  directly 
without  using  the  vldlcon  and  the  scanning  circuits,  Hence 
we  have  no  practical  way  of  calibrating  the  external  and 
Internal  models  separately,  A  separate  calibration  can  be 
done  by  a  cut-and-try  method  as  follows!  Guess  the  values 
for  the  parameters  of  one  mode)  (say  th#  external  model), 
using  these  values,  calibrate  the  parameters  of  the  other 
mode)  (the  Internal  model),  Then  using  the  new  values  of  the 
parameters  0f  the  first  model  calibrate  again  the  second 
model  and  so  on  till  some  convergence  condition  Is 
satisfied,  This  method  was  tried  by  Sobe|  and  was  found  to 
be  time  consuming  and  Ineffective,  Although  the  time 
needed  for  each  Iteration  Is  relatively  short,  (since  a 
smaller  number  of  parameters  is  handled),  the  number  of 
Iterations  needed  was  Urge, 

The  calibration  process  developed  by  Sobel  used  the 
full  <external  and  Internal)  model  and  Is  described  below: 

(a)  An  object  (usually  a  black  l  shaped  book  end)  of 
known  size  l3  put  On  the  table  at  a  known  position,  A 
picture  of  the  object  Is  stored  with  the  readings  of  the  pan 
tlit  and  focus  pots  and  the  location  of  the  object  vertices 
In  the  t.c.s,  This  process  Is  repeated  10  to  2?  times  for 
various  locations  of  the  object  on  the  table  and  various 
locations  of  Its  Image  In  the  Image  plane, 

(b)  An  edge-follower  Drogram  and  a  line-fitter 
program,  which  are  aiso  modules  of  the  hand-eye  system,  are 
run  on  each  picture  stored  and  the  location  of  the  vertices 
of  the  object  in  the  l,c,s,  Is  determined, 

(c)  A  guess  of  the  values  of  all  the  parameters  of 
the  full  model  Is  made,  Using  these  values  and  the  pot 
readings  associated  with  each  olctupe,  the  transformation 
from  t.c.s.  to  l.c.s,  Is  computed.  Using  the  transformation 
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and  the  looatlons  of  the  veptloes  In  the  t.c.s,  stored  «'th 
each  Dlotur*,  a  predlotlon  of  the  vertloes'  locatlors  in  the 
I  ,0,3,  Is  computed,  The  prediction  Is  then  compared  to  the 
I ,c,it  looatlons  obtelned  In  step  <b)  and  e  root  mean 
Square,  <RMS),  dlstanoe  error  Is  computed,  averaged  over  all 
vertices  end  a|l  pictures, 

(d)  Step  <c>  Is  repeated  while  the  values  of  the 
parameters  are  changed  so  as  to  make  the  RMS  error  sma||er 
In  each  Iteration,  Two  k|ndr,  of  parameter  minimization 
routines  ar«  used.  Neither  needs  to  explicitly  compute  the 
values  of  the  first  or  seoond  derivatives  of  the  »MS  error 
with  respect  to  eaoh  parameter  of  the  model, 


The  calibration  process  Is  done  separately  for  each 
lens,  The  R,M,S  error  that  Is  achieved  when  20  pictures  are 

used  Is  1  to  2  raster  units, 


The  values  of  some  of  the  parameters  differ  slightly 
*hen  calibrated  for  different  lenses,  although  they  ere 
known  to  be  Independent  of  lens  type.  For  example; 
P1,KX,PPOT0, TPOTD.  This  points  to  another  advantage  of 
calibrating  the  whole  set  of  parameters  together:  there  Is 
enough  "freedom"  to  compensate  for  changes  in  some 
parameters  which  depend  on  the  lens  type,  but  were  not  taken 
Into  account  In  the  model. 


Since  the  transformation  equations  do  not  use  t 
lens  focussing  equation  but  only  the  magnification,  t 
model  used  by  $obej  has  a  scaling  variable  "fmy"  which  c 
be  defined  In  the  terms  of  our  model  as:  fmy5foc»KY  and 
parameter  MART  defined  by:  MART=kX/KY.  Also  two  cth 
parameters,  FPO^B  and  FPOTD,  ar»  used  Instead  of  foCIEn0  a 
FOCLENG  to  convert  the  focus  oot  reading  directly  to  FM 
_ _ -  ...»h  by  Sobe  |  has  16  D&pametersi  PI,  p2,  P 


Hence  the 
DPI,  0P2, 
SWING, 


mode)  used 
DP3,  PPOT0, 


MART,  PP1,  PP2, 


PPOTO,  TPOT0,  TPOTD,  FPOT0,  FPOTD, 


4,3  Calibration  updating,  general  notes 


mods!  was  obtained  by  Sobel,  we 


used 
c  ,  t ,  c  , 
U  I  t ,  c  . 
errors 


:J!n5!j'ddi«;itiri^cR.oiir'3  tr?ind  the 


A f  t ® r  a  ca 

the  corner*1'  -  —  -  —  ■ 

of  some  points  on  the  table,  They  differed  from  the 

of  the  same  points  by  about  0,2  Inch,  These 

are  2  to  4  times  larger  than  that  corrspondlnc  to  an 


error  of  1  raster  unit  on  the  Image  plane  <the 
correspondence  depends  on  the  pan  and  tl It  angles)  auoted  by 
Sobel,  This  result  Is  resonable  since  the  errors  ccmcutfd 
by  Sobol  were  for  Precisely  the  points  used  In  the 
calibration  process,. 
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It  was  not'd  th#t  »ft#s  some  time  (on  the  Ofder  of  • 
M...  or  u<,>  tn«  MrfMMxn  #*  «od»i 

tNa  trrflri  or*w  to  U  to  t.i  »«oh,  So»#  » [ 
ir«  not  fliofliMod  OQpr'othr  **¥  npf#,  Oh*  ri»#d*  *  *Lll!  t! 
th«  Mho  I#  ck  |  ItorptloA  oroa#M  lY*f*  f*w  d#M,  This  f*"*«  U 
Siry  tlio  cai*u*ln$  UiPUt  M  to  49  nlnutu  n*  ; 

tlK>  *nfl  T.p*  *"  ■Itopnitlo*  l<  to  find  the 

sApOmotiri  that  hoY*  phenpod  ifid  oodeti  lf»#»  onUf, 
uodttlno  of  «ono  DtriHIOfl  towld  olio  Dirt  I  k  1 1  J  «<iwo*hkkt* 
tit  gningu  jn  #th#r  |ir«Ht«Pli  Hopefully  thli  prow* 
Uhl  b«  muon  iboftop,  Anaf  it  pom*  min/  *■ 

■  ineo  thi  n|  it  Ini  oilUrotlohi  otthoufh  not  flood  inouflh 
I^oort,  >i  suHlolont  to  Itriot  thi  pointing  of  thi 
i  ai.,  _  L  a<||  m.nh  fgp  thi  Iflofll  0*  thi  v#rt1fl*l  Ui  I  n0  thi 

”«r:nid^  SiJ.rlSio  in  Ch.ot.p  3  .11  und.r  .«««. 

contro I  with  nb  minuil  Intervention.  This  solution  was 
pursued  and  Is  desorlbid  fallow  and  In  subsipuent  sections. 

Ht  can  divide  |ho  o*rtm*tiri  Into  thi-e*  «r#L0fl 

Ut  Parifflittri  whloH  depend  on  tht  nnyaUii 
dlmifialOhi  of  thi  aoeimfrliri  PX,  P**p3  »Opi*0f  Zj  D*3| 

{ t)  ]  Parameter*  whloh  Open#  Oh  t^i  eoi r k 1 1  Oh 

electronic  cl  row  1  til  SWjNGjPPI, PP2,KXi kt ,  i  , » 

(cl  P*ra**t*rf  wn I oh  d»Q*nct  on  *.ic,r  cal 
htublrtlPk  .Jeh  Ik  th*  potl  end  the  A/D  POnV|f*#r  #0**' 

*r!:£5£ 

6,  Sdb.1  dio-d  Ob  Win  “*  le 

chanatif litlce  and  th»  iliotronfa  eoinnlna  olroult*. 


Th.  fErit  srouo  0*  o*r  imitir »  1  fl  i'"*1 
ifl*n*ot1bli  to  ohoPfli  with  tleo.  Thp  olhor  two  ar*  JJJIJ 
■uioiof  lbl*  to  ehonio,  of  tho  lOodtid  flroUP  pifpnetipo,  SWING 
iwhloh  I  a  about  1  Pifl.l.  PP1  ihd  pp®  “111  h*Yi  on  y  aeoond 
_  af  f act  on  th"  koaurkOY  of  qoi I tl Oh  orod lot  on|i  #*■" 

In  the  cue  of  SWING, 


I  io*t 

nor  a 


KX#  xy  and  thi  third  a  roue  OirdffOtin  hivi  |  j  ml  I  *r 
affitt*  Oh  the  ooeUFkcY.  Thi  acpupa&y  la  war#  J,n* 
the  Ohinai*  In  th#  o*n  «hO  tilt  Pit  oonalmti  than  to 

Imv.  *r  «  “•  st  ••  ?;n  “  l:0 ; 

f  c  I  Lowing  iiompln  *  ohirm*  U  PPflTI  o\  1*  of  \t* 


fcllowlnfl  iPainoltt  *  pnen^*  in  rr,i,  ^ ; 

rand«.  whlab  To  At  loktt  *1  diB(|  H  I  kTiU>Ihinhiri!r  1* 

ociltlon  r.f  «,A  Tnoh  At  a  point  OP  th*  tabU  «n!qh  I 
Sftohlk  t  r  on  th»  Imi  oinlir  U  tYoUM  dlotonoel.  A  0h*nfli 
■  I.  K*  v|H  siuM  #  poo  1 1  Ion  iffOf  of  I.!  Tnod  undir 
thk  lino  oonfll tlenf  *■  tpovi  If  th#  1«aO*  of  th#  point  IP  M 
III  Sordid  of  th#  f  p  rm#  t  thi  a  orrar  will  d*ar#*ip  itnurlY 
*lth  tho  fllitiPei  «f  thi  Inaflo  of  thi  point,  fro#  th#  point 
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Because  of  the  above  considerations  It  was  decided 
to  start  with  the  develoDment  of  an  updating  scheme  for  the 
third  group  D*r*m«ters#  first  the  pan  and  tilt  rot  constants 
and  then  the  focus  pot  constants, 

4,4  Pan/tl|t  head  calibration  updating 

We  have  two  routines  to  update  the  calibration  of 
the  pan  and  tilt  pots,  One  of  them  uses  the  corners  cf  cubes 
placed  on  the  table  at  Known  locations,  This  routine  Is  not 
as  completely  automatic  as  we  wish  It  to  he  since  the  cube? 
are  manually  placed  at  those  locations.  It  alms  at  reducing 
the  differences  between  the  coordinates  of  a  point  In  the 
t,c,s,  as  approximated  by  the  user  and  the  approximation 
given  by  the  c,t,c, 

The  other  routine  uses  the  arm  Itself  Inst rad  of  the 
cubes.  Slnce,  in  general#  the  Image  of  the  hand  at  the  end 
of  the  arm  ls  still  too  oompjlcated  for  the  vision  programs 
to  analyze,  a  blaok  rectangle  was  painted  on  each  of  the 
fingers  (the  hand-mark),  Also#  the  hand  Is  rotated  so  that 
one  finger  Is  facing  the  eamara#  hiding  the  other  finger  and 
parts  of  the  hand#  <see  Figure  28(a)  and  compar«  with  Figure 
28(b)),  If  the  calibration  has  not  changed  too  much, 

we  can  ask  the  corner-finder  of  Chapter  3  to  search  for  the 
corner  of  the  hand-mark  without  Its  getting  confused  by 
other  features  of  the  arm,  Demanding  a  match  of  form  and 
relative  Intensity  helps  the  corner-finder  to  recognize  the 
hand-mark.  Since  there  are  no  other  objects  on  the  table 
when  the  calibration  updating  programs  are  run,  we  can  let 
the  corn#r-f I nder  search  In  a  fairly  large  area  and  so 
handle  |ar5e  parameter  changes,  We  ar*  limited  however  in 
one  direction  by  the  presence  of  the  upper  part  of  the  hand, 

By  using  the  arm  we  achieve  two  goals:  automating 

the  process  and  reducing  the  coordination  errors,  In  effect, 
w#  Assume  her®  that  the  deterioration  of  the  pan  and  tilt 
parameters  Is  the  only  cause  of  the  coordination  errors.  (We 
rrake  a  similar  assumption  for  the  focus  Dot  In  the  next 
section),  We  uDdate  the  paramaters  to  minimize  the  measured 
coordination  errors#  (Actually  we  do  not  measure  them  but 
only  their  effect  on  the  location  of  the  hand-mark  corner  In 

the  I , c , s ,  (see  st®D  (o)  below  and  Appendix  A). 

1 

The  updating  Is  done  In  the  following  stecs: 

(a>  The  camera  Is  centered  and  focussed  on  the  next 
position  of  the  arm  (or  tha  next  cube),  Centering  is 
performed  to  ascertain  that  the  image  of  the  corner  sought 
Is  In  the  frame  and  that  the  effect,  of  the  changes  In 
Internal  model  parameters  Is  minimized.  The  precision  of 

the  centering  process  does  not  affect  the  updating, 
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(0)  HAND  FACING  THE  CAMERA 


(b)  HAND  IN  GENERAL  POSITION 


Fig.  28.  THE  HAND  AS  SEEN  BY  THE  CAMERA. 
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<b)  Th«  oamera  transformation  Is  computed  for  the 
naw  (oantapad)  position,  Tha  ooopdlnatas  of  the  I  mage  vertex 
and  the  fopm  of  tha  Image  copnap  ape  than  computed  uslno 
this  transformation, 

(c)  Tha  copna r- f  I  ndap  Is  celled  and  given  the  above 
I nf opmat Ion,  Whan  usImq  tha  cubes*  we  chose  dark  colored 
cubes  on  a  light  background  or  vlca-vapsa,  When  using  the 
hand  the  corner  of  th*  hand-mark  Is  sought,  In  both 
cas«s  a  relative  Intensity  match  Is  demanded  lr<  addition  to 
form  match  as  mentioned  before,  If  the  corner  Is  not  found, 
as  happens  occasionally  when  the  calibration  has  not  boen 
updated  for  some  time,  the  routine  goes  back  to  step  (a)  and 
to  the  next  cube  or  arm  position, 

After  the  corner  Is  found  the  error  between  its 
aotual  and  Predicted  looatlon  In  che  l.c.s.  Is  computed, 
This  error  Is  used  In  two  waysl  First,  we  assume  that  at 
least  part  of  the  error  Is  systematic  and  ence  we  add  the 
error  to  the  ppedloted  location  of  vhe  next  corner  before 
calling  the  copnep-f I nder ,  This  turned  out  to  be  a  real 
winner,  Second*  we  compute  what  the  pan  and  tilt  angle 
should  be  |n  order  that  the  actual  and  predicted  locations 
wl|l  coincide,  we  record  these  ancles  with  the  actual 
reading  of  the  pan  and  tilt  pots, 

(d)  After  data  has  been  collected  from  a||  the  cubes 
or  arm  positions  (less  the  cases  where  the  corner  has  not 
been  found),  two  8 1  ra » 0ht  M  ne  a  are  fitted  to  two  s«ts  of 
points  with  the  pan  and  tilt  angles  as  abscissas  and  the  can 
and  tilt  pot  readings  as  ordinates,  The  coefficients  of  the 
two  lines  are  the  updated  pots'  parameters  for  can  and  tilt, 

4,5  Focus  calibration  updating 

As  noted  before,  Sobers  model  combined  KX  and  ky 
with  foe  because  there  was  no  way  to  distinguish  between 
them,  One  reason  Is  that  when  the  measurements  were  taken 
for  the  calibration,  the  camera  was  focussed  on  the 
calibration  object  as  a  whole  and  not  on  any  particular 
point  which  could  be  used  to  calculate  separately  the 
coefficients  fop  the  focus  pot,  On  the  other  hand,  we  need 
exactly  those  separated  coefficients  In  order  to  calculate 
the  focus  pot  setting  needed  to  focus  the  camera  at  a 
particular  range, 

In  this  case  also,  either  a  set  of  cubes  placed  on 
the  table,  op  the  arm  Is  used,  The  Positions  are  chosen  to 
present  a  verlety  of  ranges,  The  updating  is  done  in  the 
following  steps* 
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(a)  The  camera  Is  oentered  on  the  next  position  of 
the  arm  (or  the  next  oubo), 

(b)  The  oamers  le  foousied  on  the  oorner  of  the 
hand-mark  on  the  finger  (which  Is  made  to  directly  face  the 
camera),  op  on  the  next  cube,  by  moving  the  vldtcon  ( I • e 
changing  foe),  In  the  beginning  the  fbousslng  was  done 
manually,  Then  we  added  an  automatlo  fooueslng  routine  whloh 
was  developed  by  J,M,  Tenenbautt  C283,  (See  Appendix  B),  In 
the  latter  oeee  the>  oorner-f Inder  tc  celled  first  to  exaotly 
locate  the  Oorn«r  before  the  automatic  fooueslng  routine  Is 
called,  The  foous  pot  reading  Is  then  recorded, 

(c)  The  range  from  the  lens  oeriter  to  tbs  above 
corner  Is  computed,  and  then  using  the  foous  equation  the 
corresponding  value  of  foo  Is  oomouted  and  recorded, 

(d)  After  all  data  has  been  ool  looted  a  straight 
line  Is  fitted  to  the  set  of  points  with  the  oompgted  values 
of  foo  as  absojssa  and  the  foous  pot's  readings  as  ordinate, 
The  ooefftolents  of  this  line  are  <;he  updated  focus  pot 
parameters, 

It  was  noted  before  that  when  calibrating  for 
different  lenses  the  parameters  were  not  foroed  to  take  the 
same  values  even  If  there  was  no  reason  for  them  tc  be 
different  foP  different  lenses,  In  the  case  of  the  foous  pot 
parameters  we  have  to  stray  fro*  this  principle,  The  reason 
Is  that  for  the  small  fooal  length  lens,  1  Inoh,  the  depth 
of  field  Is  large  enough  so  that  we  can  not  fit  |  line  tjo 
the  readings  with  enough  oonfldenoe,  Hence  ws  use  the  fact 
that  the  slope  of  the  line  hss  Indeed  the  same  value  for  all 
lenses  and  only  the  bias  changes,  Thus  a  long  focal  length 
lens  (3  Inch)  Is  us«d  to  oaHbrate  the  slope,  For  the. 
smaller  focal  length  lenses  only  the  bias  Is  updated  by  the 
above  prooess  using  the  slope  already  obtained, 

Since  we  do  not  yet  have  a  good  external  way  to 
measure  the  quality  of  focussing  ,  we  have  to  r»|y  either  on 
the  self  scoring  automatic  focussing  module  or  on  a  human 
obstrver  and  the  monitor  to  Judge  that  the  updating  process 
Is  adequate, 

4,6  More  remarks  about  updating 

Whan  the  degradation  of  the  parameters  Is  so  ! • r ge A 
that  the  corn«r-f Inder  cannot  find  the  hand-mark,  eltheP 
beoause  It  Is  gut  of  focus  (|f  the  degradation  Is  of  the 
foous  pot  parameters)  or  bsoause  It  Is  out  of  the  search 
eoaos  (If  the  degradation  is  of  the  pan  and  tilt  not 
parameters),  Or  botho  In  this  case  we  use  the  manual  mode  of 
the  update  Programs, 
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The  manual  mode  for  the  focus  pot  updating  orogram 
was  described  above,  In  the  pan  and  tilt  pots  update  orogram 
we  have  a  provision  for  manually  positioning  the  window  on 
the  desired  corner,  Usually  this  has  to  be  done  only  for  the 
first  position  of  the  arm  because  of  the  "Instant 
adaptation"  desorlbed  In  step  (e)  of  Section  4,4, 

If  the  hand  mark  Is  out  of  focus,  the  focus  updating 
routine  (In  manual  mode)  has  to  be  called  first,  After  that 
the  oan/t I  1 1  updating  routine  can  be  called,  Then  we  should 
call  the  focus  updating  program  once  again  to  make  the  model 
iror®  accurate  (this  time  It  can  be  called  in  automatic 
mode),  The  above  cycle  can  be  repeated  two  or  three  times 
until  no  more  jmprovments  can  be  made, 

There  are  some  alternatives  to  the  use  of  routines 
which  are  executed  specifically  for  the  purpose  of  updating* 

(a)  The  coordination  errors  (i.e  differences  between 
the  c.t.c,  and  h,t.o,>  which  are  detected  during  regular 
operation  of  the  hand-eye  systsm  are  stored  Indexed  by  the 
place  over  the  table  where  they  occurred,  Then  they  are  used 
subsequently  for  this  region  of  the  table,  Interpolation 
between  locations  for  which  the  coordination  errors  were 
stored  should  be  considered,  A  similar  soheme  is  used  In  the 
visual  feedback  tasks  to  augment  the  updating 

done  with  the  routines  desorlbed  In  this  chapter, 

(b)  The  values  of  the  errors  are  used  to  update  the 
model  each  time  an  error  above  some  threshold  Is  detected, 
The  correction  term  should  be  weighed  appropriately, 
otherwise  the  model  wl||  "thrash". 

(c)  The  values  of  the  errors  are  stored  regularly. 
If  a  degradation  of  the  mode)  Is  detected  (based  on  more 
than  one  ePror),  ,then  all  the  stored  errors  are  used  to 
update  the  mode  I  ■ 

For  the  pan/tllt  updating,  the  equations  for  (b)  and 
(c)  are  more  complex  than  those  described  In  Appendix  A  for 
the  method  of  Section  4,4,  since  the  errors  are  not 
necessarily  observed  near  tbe  piercing  point  of  the 
prlnotpal  py. 

The  advantage  of  the  alternative  methods  Is  that  no 
special  Programs,  Or  the  the  to  run  these  programs  is 
needed,  The  disadvantage  Is  that  we  have  to  make  sure  that 
the  coordination  errors  were  detected  and,  measured  with  high 
reliability,  This  Is  not  a  problem  with  the  special  purpose 
programs  since  the  environment  Is  very  simple,  b u it  could 
cause  trouble  during  regular  operation,  where  the 
environment  Is  often  cluttered, 
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CHAPTER  5:  VISUAL  FEEDBACK  TASKS 
5.1  Introduction 


5,1,1  Genepai  not*s 

The  purpose  of  the  visual  feedback  (v.f.b.)  tasxs  Is 
to  Increase  the  precision  of  the  manipulations  done  In  the 
hand-eye  system.  The  feedback  currently  does  not  take  Into 
account  the  dynamic  aspects  of  the  manipulation  (see  father 
discussion  in  Section  5*3>'2)> 

The  tasks  are  carried  out  In  the  context  of  the 
general  tasks  that  the  hand-eye  system  can  current  y 
perform,  l,.e,  the  reooflnltlon  and  manipulation  of  s  mole 
planar  bounded  objects,  whose  faces  are  uniformly  painted 
(without  texture)  with  various  colors,  on  a  background  a  so 
of  uniform  shade  (usually  dark),  The  lighting  Is  supplied 
either  by  the  regular  fluorescent  overhead  lighting  wh.ch  is 
somewhat  but  not  completely  diffused,  or  by  three  Quartz 
Iodine  lames  which  can  boost  the  usable  intensity  dynamic 
range  but  also  create  harsher  shadows  which  are  harder  to 
cope  with, 

The  manipulations  that  we  sought  to  make  m^e 
precise  with  the  addition  of  visual  feedback  are  graso.ng 
placing  on  the  table  and  stacking,  From  now  on  when  we 
mention  these  three  operations,  It  will  be  meant  that  they 
are  done  under  v.f.b,  control. 

The  precision  obtained  Is  better  that  .1  [neb.  This 
value  should  be  Judged  by  comparing  It  with  the  limitations 
of  the  system.  The  resolution  of  the  imaging  system  with  the 
2  I noh  |ens  is  1  mrad,  which,  at  an  operating  range  of  30 
Inch  corresponds  to  ,03  Inch.  The  resolution  of  tne  arm 
position  reading  (lower  bit  of  the  A/D  converter  reading  the 
first  arm  Joint  potentiometer)  Is  a|so  ,03  inch,  Cut  the 
noise  In  the  arm  position  reading  corresponds  to  ,05  inch, 
when  we  tried  to  achieve  preolslon  of  ,05  inch,  the  feedback 
loop  was  executed  a  number  of  times  untlll  the  rrors 
randomly  happened  to  be  be|ow  the  threshold, 

The  rest  of  this  section  Is  devoted  to  tne 
discussion  of  some  general  Problems  pertaining  to  all  three 
tasks,  The  next  three  sections  describe  In  detail  the  three 
tasks  as  Implemented  for  cubes,  Since  the  termination  of  the 
research  described  here,  the  caoablllty  of  the  program  has 
bee  extended  by  R,  Davis  to  handle  other  objects  (wedges  and 
slabs)  and  one  mor®  task  of  Inserting  objects  into  holes  of 
the  same  form. 
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5,1,2  Dynamlo  fatd&aok 


The  Question  of  whether  the  visual  feedfcaok#  or  In 
general  the  response#  Is  dynamlo  or  not»!s  sometimes  more 
semantic  than  real,  What  the  person  asking  the  Question 
means  In  this  case  Is#  does  It  seem  to  pe  continuous?  The 
question  then  Is  really  that  of  cyole  time  or  sampling 
period,  a  cycle  time  of  20  msec  will  suffice  to  fool  the 
human  eye  so  that  the  response  w|l|  be  called  "dynamic”, 
Since  the  computations  needed  and  the  computing  rower  that 
we  now  have  cause  the  length  of  the  cycle  time  to  be  several 
seoonds#  no  attempt  was  made  to  speed  I t  up  by  programming 
tricks#  use  of  machine  language,  eto.  With  this  cycle  time 
the  movement  of  the  arm  actually  stops  before  we  analyze  the 
situation  again#  so  that  we  do  not  have  to  take  Into  account 
the  dynamic  aspect  of  the  error* 

In  addition  to  oomputlng  power#  the  vldlcon  camera 
also  presents  some  limitations  to  faster  response.  The  short 
time  memory  of  the  vldlcon  which  halos  us  (persons)  to  view 
the  TV  monitor#  will  "smear"  a  fast  moving  object,  If  the 
scene  Is  bright  enough  a  shutter  oan  be  used,  If  the  vision 
can  be  made  sufficiently  fast  and  acourate,  the  control 
program  C183  currently  used  to  run  the  arm  dynamically  could 
be  expanded  to  Incorporate  visual  information, 

5,1,3  Hand  In  view 

As  already  mentioned,  one  of  the  characteristics 
that  distinguishes  our  visual  feedback  scheme  |3  that  the 
analysis  of  the  scene,  to  detect  the  errors  to  be  corrected, 
Is  done  with  the  hand  (which  is  still  holding  the  object)  In 
the  scene,  In  the  grasping  task  the  presence  of  th«  hand  Is 
Inevitable,  jn  other  tasks,  for  example  stacking,  balno 
able  to  analyze  the  scene  with  the  object  still  grasped 
helps  to  correct  the  positional  errors  before  they  become 
catastrophic  (e,g  the  stack  falls  down),  Also  some  time 
Is  saved  since  there  Is  no  need  to  release  the  object#  move 
the  arm  away,  bring  It  baok  and  grasp  the  object  again,  We 
pay  for  this  flexibility  with  Increased  complexity  of  the 
scene  analysis. 

The  difficulty  Is  lessened  by  the  fact  that  the 
hand-mark  (which  Is  used  to  Identify  the  hand)  was  known 
form  and  relative  Intensity  which  he | o r  to  looate  It  In  tne 
Image,  In  the  grasping  task  the  abijlty  to  recogniza  the 
hand  Is  necessary,  The  task  Is  essentially  to  position  the 
hand  at  fixed  looation  and  orientation  relative  tc  the 
objeot  to  be  grasped, 
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Also,  we  have  found  It  to  our  benefit  to  locate  the 
hand  first  In  the  otn*r  tasks  alio.  After  the  hand-mark  Is 
found,  we  use  Its  location  to  predict  more  accurately  the 
locations  In  the  Image  of  the  ed9es  of  the  object  held  by 
the  hand,  We  can  do  this  since  the  object  was  gresoed  w  th 
visual  feedbaok,  and  therefore  Its  position  and  oriental  on 
|n  the  h,c,s,  are  Known.  {We  also  take  Into  account  the 
remaining  errors  of  the  grasping  task).  We  use  tne 
eoo  r h I  nates  In  the  h.c.s,  to  f I n  d  t^e  h.t.r.  of  the  ed903  , 
From  the  h.t.c,  we  predict  the  Image  c00rdlnates.  We  c0rr®ct 
the  prediction  by  the  difference  between  the  predicted  and 
actual  location  of  the  hand-mark  found  In  the  Image,  This  is 
the  right  correction  since  the  Prediction  of  the  hand-mark 
location  In  the  Image  was  obtaineo  by  exactly  the  same 
calculations.  (This  Is  vepy  similar  to  the  "instant 
adaptation"  described  In  Section  4,4). 

Moreover,  after  the  hand-mark  has  been  found,  it 
will  not  be  confused  with  other  edges  In  the  scene. 

5,1,4  Inference  of  three  dimensional  Information 

In  the  pan  and  tilt  calibration  updating  orograms  we 
assumed  for  convenience  that  the  coordination  errors  were 
caused  solely  by  deterioration,  over  time,  of  the  pan  'And 
tMt  pot  constants,  By  executing  the  calibration  updating 
program  we  removed  part  of  the  coordination  errors  (mainly 
the  part  that  had  systematic  characteristics  over  various 
locations  In  the  t.C.s.), 

In  the  v.f.b  tasks  we  assume,  again  for  convenience, 
that  the  remaining  coordination  errors  (which  will  be  called 
simply,  errors)  are  caused  by  the  arm. 

These  two  assumptions  .are  made  for  covanlence  out 
really  they  are  eoulvalent, 

As  explained  Iri  Section  4,1,2,  the  c.t.c. 
approximation  Is  used  for  reference,  Since  we  are  using  only 
one  camera  (one  view),  we  cannot  measure  d|ractly  even 
differences  of  locations  of  two  neighbouring  points.  Hence 
the  three-dimensional  (3-D)  Information  has  to  be  Inferred 
from  the  two-dimensional  <2-D)  Information  available  in  the 
Image  and  some  external  Information, 

One  assumption  that  ccn  be  made  concerning  points  on 
the  hand  or  on  an  object  held  by  the  hand,  is  that  the 
positional  error  of  the  arm  in  a  particular  direction,  for 
example  height  above  the  table,  Is  much  smaller  than  the 
errors  In  the  other  directions,  This  assumption  can  then  be 
used  to  Infer  the  errors  In  the  other  directions  from  the 
2-D  Image,  If,  however*  such  an  assumption  Is  made 
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Incorrectly#  the  error  In  the  direction  which  we  assumed 
errorless  w|||  cause  errors  of  the  same  order  of  magnitude 
to  be  Inferred  for  the  other  directions.  We  do  not  have  any 
reasons  to  Justify  this  assumption  In  our  system  and  other 
external  Information  has  to  be  used,  . 

We  assume#  however#  that  the  main  errors  are  In 
position  and  not  In  orientation,  A  small  error  In 
orientation  when  multiplied  by  the  lenflth  of  the  openlnq 
between  {he  fingers  will  generate  a  position  error  of  +he 
hand-mark  which  oan  be  negleoted  relative  to  the  position 
errors  of  the  arm.  For  similar  reasons  we  also  assume  that 
the  error  In  measuring  the  distance  between  the  fingers  can 
also  be  neglected, 

”he  external  Information,  which  Is  used  in  the  v.f.b 
tasks#  I v  supplied  either  by  the  touch  sensors  on  the 
finger^  or  by  the  fact  that  an  object  Is  resting  on  the 
table-top  or  on  another  object  of  known  dimensions  (this  is 
the  famous  3uoport  hypotehsls  mentioned  |n  Section  4,3.  ,2), 
The  touch  sensors  help  us  to  determine  the  plane  containing 
the  hand  mark  from  the  known  position  of  the  touoheo  object, 
The  supper t  hypothesis  gives  us  the  plane  containing  the 
bottom  edges  of  the  too  object.  More  Implementation  details 
are  given  with  the  descriptions  of  the  various  v.f.b  tasks. 

5,1,5  Hand-Eye  coordination. 

To  facilitate  the  scape  analysis  we  use  here  one  of 
the  methods  fer  calibration  updating  mentioned  already  in 
Section  4,6.  We  store  the  coordination  errors  found  during 
regular  operation  of  the  system  (specifically,  carrying  out 
v , f , b  tasks)  and  use  them  subsequently  as  described  below, 
we  Justify  th's  by  the  as«umDtlon  that  no  maJor  change  of 
the  arm  and  camera  hardware  Is  oclng  to  occur  during  the 
short  time  of  operation,  If  needed  we  can  add  the  time  that 
the  errors  were  stor«c  and  find  and  store  them  again  after 
they  become  too  old,  we  do  not  currently  attempt  to  use  the 
stored  ocordjnation  errors  for  actual  urdatind  of  the  camera 
or  arm  models# 

Actually  wa  do  not  store  the  coordination  error; 
themselves  but  two  quantities  clo°ely  related: 

(a)  Aftflr  an  object  Is  grasped  and  then  released, 
the  h.t.c,  of  the  reference  point  of  the  hand  (midpoint 
between  the  touch  sensors)  Is  stored  with  the  object  30  that 
the  next  time  we  want  to  grasp  the  object#(or  stack  on  It)# 
we  use  the  h.t.c.  rather  than  the  Cit.c.to  position  the  arm. 
However#  If  the  object  was  neeep  grasped  before#  the  c.t.c, 
are  used,  The  v.f.b  Is  used  In  both  cases  (and  not  only 
the  seoond )  to  check  and  correct  the  grasping.  The  task 
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Is  simpler  If  the  h.t.c.  are  already  known,  because  the 
repetition  errors  of  the  arm  servo  are  smaller  than  the 
coordination  errors, 

This  last  characteristic  also  explains  the  following 
fact*  The  Initial  error  I n  stacking  Is  smaller  If  the 
supporting  object  has  already  been  grasped  once  by  the  hand 
and  hence  Its  h,t,c,-  already  stored,  In  this  case  the  main 
part  of  the  coordination  error  is  already  found  and 
corrected  by  the  v,f,b  during  grasping,  On  the  other  hand, 
If  the  supporting  object  was'  never  grasped  before,  the 
coordination  error  has  to  be  corrected  by  the  v.f.b  control 
during  the  stacking  which  Is  more  complex  and  difficult  than 
the  control  of  QrAspIng,  If  really  needed,  the  object  to  be 
stacked  upon  can  actually  be  grasped  first,  (with  v.f.b),  to 
find  its  h,t,c,»  and  then  released  to  continue  with  tne 
stacking  task. 

(b)  Before  an  obieot  Is  to  be  grasped  or  stacked 
upon,  the  arm  Is  positioned  above  the  object  and  the 
hand-mark  ls  sought  as  was  done  In  the  pan  and  tilt 
calibration  updating,  The  hand  Is  positioned  high  enough 
above  the  object  so  that  the  corner-finder  does  not  confuse 
the  hand-mark  with  the  object.  After  the  hand-mark  Is 
found,  the  dlfferenoe  between  the  coordinates  In  the  i.c.s, 
(Image  coordinate  system)  of  the  predicted  location  of  the 
hand-mark  and  the  location  where  It  was  aotua||y  found  ls 
stored,  This  Is  the  same  Quantity  stored  In  the  pan  and  tilt 
calibration  UDdatlng  Program  for  the  "Instant  adaptation". 
The  same  Is  done  for  the  place  on  tne  table  where  an  object. 
Is  going  to  be  p I aced, 

The  table  is  divided  Into  4  Inch  squares,  (there  »ro 
100  sauares),  and  the  corrections  are  stored  with  the  souare 
over  which  they  were  found,  When  we  Subsequently  look  for 
the  hand-mark  over  this  part  of  the  table,  the  stored 
alfferencas  are  used  to  correct  the  prediction,  Since  we 
now  have  a  corrected  prediction,  the  dimension  of  the 
window,  or  the  search  space  used,  can  be  made  smaller  , 

Each  time  that  the  hand-mark  Is  found  again,  the 
clfferences  between  predicted  (before  correction)  and  actual 
locations  lr  the  Image  are  also  used  to  update  the  stored 
correct! ons » 

Not®  that  we  cannot  Infer  the  three  dimensional 
coordination  ePror  from  the  two  dimensional  difference,  as 
measured  above,  since  there  Is  no  external  Information  that 
can  be  used  in  this  case, 

b  ,  l ,  6  "State  of  the  world"  updating 
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( WMli )  An  Indication  of  whether  the  object  Is  stable 
and  hence  can  be  released. 

( WM12  )  An  Indication  of  whether  the  object  Is 
currently  held  by  the  hand, 

For  each  face  of  a||  objects! 

(WM13)  An  Indication  of  whether  the  face  Is  visible 
to  the  camera,  (For  cubes#  the  sine  of  the  angle  between 
the  line  of  sight  of  *he  camera  when  centered  on  the  cube 
end  the  normal  to  the  "most  visible"  vertical  face, 
described  In  the  next  section,  The  visibility  of  other  faces 
or  edges  ar®  computed  from  this  Information), 

<WM1<)  The  normal  (In  o,t,e.)  to  the  face.  (For 
cubes,  the  normals  to  the  two  vertical  visible  faces). 

Som®  of  the  components  are  redundant,  For  example, 
the  normals  to  the  faces  can  be  comDUted  from  the  prototype 
Information,  ('In  WM5),  and  orientation  (In  WM6),  This  is  an 
example  to  the  genera|  problem  of  "store  vs  recompute"! 
when  to  store  enough  redundant  Information  to  speed 
execution,  whan  to  store  Information  only  aftor  it  was 
needed  once,  for  possible  subseouent  use,  or  when  to  compute 
It,  again  each  time  It  is  needed, 

We  now  proceed  to  discuss  the  problems  of  uodatlng 
the  state  of  the  world,  If  each  operation  was  totally 
reliable  we  could  update  the  state  of  the  woP|d  after 
planning  the  change,  or  after  carrying  out  the  operation  but 
before  the  Information  Is  needed  by  other  modules,  or  at  any 
time  In  between,  If  a | I  the  failures  were  of  a 
gem  I -pe r mane n t  nature,  (something  has  to  be  externally  fixed 
before  the  modules  oould  carry  on),  then  again  we  could 
update  any  t]mr,  before  the  Information  is  needed  and  quit 
altogether  at  the  first  failure. 

Overall,  the  reliability  of  the  manipulation  Is  very 
good,  However ,  from  time  to  time  the  arm  (or  hand)  w.ll 
fall  because,  for  example,  of  rare  circumstances  of  the 
time  sharing  system,  or  transients  In  the  hardware,  Most  of 
thesj  failures  do  not  have  catastrophic  effect?  and  the  arm 
slmoly  stops  moving  and  Indicates  the  source  of  failure,  On 
some  occasions,  blindly  repeating  the  operation  Is  the  wrong 
thing  to  do,  for  examole  when  executing  an  Incrementally 
specified  movement  of  the  arm,  In  other  oases  we  would  IIKe 
to  change  or  abort  the  nanlpulstlon  after  the  failure, 
Therefore  we  need  to  time  the  updating  carefully  so  that  the 
updated  Information  wpuld  be  available  immediately  after 
failure  but  not  prematurely, 


* 
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For  the  same  reasons  we  need  both  WMfl  and  WM10*  or 
both  WM12  and  WM1#  whloh  are  used  to  give  correct 
Information  for  different  modes  of  failures*  by  not  being 
updated  at  the  same  time, 

For  example*  We  use  WM1  to  Indicate  that  the  object 
pointed  to  may  be  In  the  hand  and  therefor®  we  need  to 
release  It  before  movlnfl  the  arm  but  It  la  not  guaranteed 
that  the  obJeot  Is  firmly  grasped  and  henee  we  have  to  grasD 
It  If  we  want  to  move  It  with  the  afm.  On  the  other  hand 
WM12  Indicates  that  the  object  Is  firmly  gresped  by  the  hand 
and  hanoa  will  mova  with  I t • 

If  -*t  tried  to  Grate  iM  object  and  filled  * t  would 
update  wp*l  btcaus*  th*  drTo  (right  be  ffi*n  anoug*'  to  mo**  the 
oojeot  a  little  before  Ft  slipped  from  the  hand*  VMI2  would 
be  updated  only  after  the  grasping  Fa  successfully 
completed,  similarly,  If  w*  tried  to  release  tnt  q  c  ;  ►  c  t  and 
failed*  UKI2  would  ba  updated  but  not  WMi  "hips*  111  ba 
updated  only  after  the  r«l«ao«  rras  successfully  coma  I itf d . 

The  various  comoonants  of  the  state  of  the  world  are 
updated  at  the  right  times  so  that  the  following  rules  can 
be  observed  and  be  truly  satisfied,  Some  of  the  reasons  for 
these  rules  are  obvious*  others  will  beeome  clear  In  the 
description  of  the  various  tasks, 

(I)  The  c.t.c.  of  an  object  have  to  be  known  In 
order  to  araso  It  or  stack  on  It. 

(IP  An  objaot  Is  not  rejeased  unless  it  Is  known  to 
bs  stable  which  means  that  It  Is  rastlno  on  the  tabje  or 
supported  by  other  objact(s)  In  a  stable  position,  In  the 
(attar  oase  It  necessarily  means  that  Its  e.t.c,  are  known, 

(III)  An  objaot  Is  gnasped,  placed  on  the  table  or 

stacked  on  another  obJeot  only  |f  the  appropriate  I.c.s, 
corrections  are  known, 

(IV>  An  obJeot  that  Is  cuymortlng  another  object 
cannot  be  gasped  or  stacked  upon.  In  more  complex 
situations  when  more  that  onn  object  will  support  or  oe 
supported  by  other  objeota#  we  will  hjive  to  check  more 
complex  oondjtlonsi  can  objaot  A  support  In  a  stable 
position  an  additional  obJeot  B?  Will  tha  removal  of  object 
A  cause  any  obj8ct  to  be0oma  unstable? 

(V)  The  vicinity  of  the  location  on  the  table  where 
an  object  Is  to  be  placed  has  to  be  empty  of  other  objects, 
A  v,f,b  task  of  abutting  obje  ,ts  has  not  been  Implemented 
yet. 
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Some  of  the  rules  end  update  timing  are  simple 
because  we  currently  have  on|y  one  arm  with  one  hand  that 
can  hold  and  manipulate  only  ope  object  at  a  time,  For 
examples  When  we  have  two  arms  we  will  allow  one  of  them  to 
hold  an  object  to  make  It  firmer  even  If  It  supports  another 
object#  but  not  to  move  It#  etc. 


The  appropriate  rules  ar®  eheoked  before  each 
operation  Is  attemoted,  If  rule  (III)  Is  not  satisfied  the 
proflram  wlli  try  to  find  the  corrections  as  described  m  the 
last  section,  Failure  to  satisfy  the  rules  causes  failure  of 
thr,  v.f.b  tasks  unless  otherwise  stated  in  the  tasks 
descriptions, 


For  brevity's  sake  we  will  not  mention  fhese  facts 
again  In  the  description  of  the  tasks,  We  will  indicate  the 
points  In  execution  where  various  components  of  the  world 
model  are  being  updated. 


5,1.7  Manipulations  not  under  visual  feedback  control 

The  manipulations  described  In  this  section  are  not 
carried  out  under  direct  v.f.b  control.  They  are  used  by 
some  or  all  of  the  v.f.b  tasks#  end  described  here  In  order 
not  to  repeat  the  description  eacf  time  they  are  used.  We 
will  Indicate  their  usage  by  writing  their  names  In  capital 
letters. 


Some  movement  of  the  arm  or  the  hand  will  at  l00st 
be  attempted  In  each  of  the  following  tasks,  Hence,  one 
component  of  the  world  model  w|l|  be  changed  regardless  of 
the  degree  of  success,  We  will  mention  this  ohange  before  we 
mention  the  movement  or  the  command  to  the  arm  controller. 


UNoRASP  ^  there  Is  a  cube  pointed  to  by  WMi,  it  Is  checked 
for  stability,  If  the  cube  Is  unstable,  failure  Is  Indioated 
and  the  ta3-.  :9  terminated,  Otherwise  we  Indicate  In  WM12  of 
the  oube  that  was  grasped#  If  any#  that  It  i e  no  longer 
firrrt|y  gripped,  Then  we  send  the  arm  controller  the 
open-hand  command,  If  It  succeeds  we  update  WMB  for  the 
cube  tnat  was  grasped  (if  any;*  and  de|sL»e  the  pc  ter  from 
WMi,  The  required  distance  „etween  the  touch 
given  to  the  procedure  as  a  parameter. 


censors  Is 


In  this  manipulation  the  cube  Is  lifted  about  ,5 
Inch  above  its  support,  First#  we  Indicate  for  the  cube  In 
the  hand  that  It  Is  not  supported  any  more  (In  WM10)#  that 
It  Is  unstable  (In  WMll)  and  that  Its  c.t.o,  are  not  updated 
(In  WM7),  Then  we  send  the  arm  controller  the  command  to 
lift  the  oube,  IF  It  succeeds  we  delete  the  pointer  from 
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v» 

WM9  of  tha  supporting  cubs  <|f  any), 

PUTOOWN 

This  manipulation  pilots  a  cube  on  Its  support  using 
forct  feedback  of  the  arm  to  stop  pushing  down,  first#  a 
polntar  to  tha  cube  held  In  the  hand  Is  placed  In  WM9  of  the 
cube  placed  uoon  (If  any).  Then  we  send  the  arm  controller 
the  command  to  place  the  oube,  If  it  sucoceeds,  a  pointer 
to  the  support  Is  placed  In  WM10  of  the  oube  In  the  hand, 
If  the  cubs  was  plaoed  on  the  table  It  will  also  be  marked 
stable  (In  WMll ) , 

This  manipulation  actually  supplies  the 
justification  for  the  suppport  assumption  when  analyzino 
soenes  with  an  objeet  still  he|d  by  the  hand.  Another  way  to 
look  at  this  manipulation  Is  to  note  that  It  actually 
eliminate  the  coordination  error  In  the  z-dlrectlon  of  the 
t .  c  .  s . 

MOVETO 

This  manipulation  moves  the  arm  with  the  cube 
grasped  in  the  hand  (If  any)  to  a  location  and  orientation 
given  to  It  as  parameters,  first  the  cube  In  the  hand  Is 
marked  unstable#  It  Is  Indicated  that  Its  c,t,c,  are 
unknown  and  the  pointer  to  its  support  Is  removed,  Then  we 
send  the  arm  controller  the  command  to  move,  If  It  succeeds 
the  pointer  )n  WM9  of  the  cube  (if  any)  that  supported  the 
cube  In  the  hand  (If  any)  Is  removed,  Each  trajectory 
starts  and  ends  with  the  fingers  pointing  down  and  moving  In 
a  vertical  d| rectlon, 

To  complete  the  |lst  wg  will  describe  the  routine 
used  to  find  the  hand-mark, 

FIND. HAND 

In  this  routine  we  are  looking  for  both  corner^.  and 
not  for  only  one  as  In  the  calibration  updating  pro  am, 
since  now  the  scene  Is  comr;|cated  by  the  presence  of  other 
objects , 

The  camera  Is  centered  on  the  hand-mark,  Using  the 
camera  and  a^m  models#  the  locations  of  the  Images  of  the 
two  lower  corners  of  the  hend-mark  In  the  i'.c.s,  are 
predicted,  the  predictions  are  corrected  by  the  stored 
l,c,s,  corrections  (WM2)  If  they  have  already  been  obtained. 
Also  the  form  of  the  Image  of  the  corners  Is  computed  as  wag 
oone  In  Section  4,4,  The  predicted  width  of  the  hand-mark  In 
the  Image  Is  stored, 

Using  the  Information  computed  above#  the  right  side 
oorner  Is  sought  first#  using  the  cor nsr-f 1  Oder  of  Chapter 
3,  If  the  corner  Is  found#  the  error  oetween  Its  predicted 
and  actual  locations  Is  used  to  update  the  prediction  of  the 
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looatlon  of  the  left  s 1 de  corner  which  Is  now  sought,  I'  the 
right  corner  ta  not  found  w#  look  for  the  left  one  first 
(see  b|ook  diagram  In  Figure  29), 

This  algorithm  Is  «n  example  of  the  use  of 
Information  about  a  relation  between  two  features  to  be 
found  In  addition  to  Information  pertaining  to  each  feature 
alone, 

We  check  thet  we  found  the  corners  belonging  to  the 
hand-mark,  and  not,  those  belonging  to  a  cube  which  might 
have  very  similar  form#  by  comparing  the  distance  between 
the  oorners  In  the  Image  with  the  stored  predicted  width. 

The  la$t  computed  prediction  errors  ere  stored  In 
WM2#  with  Indication  In  WM3  |f  needed#  for  the  appropriate 
sauare  part  of  the  table, 


In  the  f o I  I  owing  desor lot  I  on  of  the  v,f,b  tasks#  If 
any  of  the  above  manipulations  or  other  operations#  like 
centering  the  oamSra#  finding  a  line#  eto,#  falls#  It  will 
cause  failure  of  the  v,f,b  task  unless  otherwise  stated, 

5,2  Griftrtnff  Silk  tCWiSP) 

The  taak  Ti  to  precisely  or* id  «  cube  of  Known 
pc&TtTorv  *nb  orientation  In  order  to  "ipvt  It  *md 
elate  It  somiwntre  elM#  or  staoh  It  on  another  Cube,  Tr.s 
precision  i*  needed  In  order  not  to  drop  the  cube  In 

trajectory  (which  c*n  happen  If  the  cube  Is  gresoto  too 

close  to  *n  edge)#  end  In  order  that  Its  position  relative 
to  the  hand  will  be  known.  This  Information  Is  used  In  the 

other  two  v ,  f ,  b  tasks,  He  try  to  anon  the  cube  on  the 

re  I  p*  |  \  nd  betwe«n  the  facts  oe^pind  I  ou  |ar  to  the  fingers, 
he  |  f  way  above  the  center  <3*  brevity, 

Net*  that  In  one  direction  (namely  perpend |ouTir  tfl 
the  fingers)  the  hand  does  Its  own  error  correcting!  When 
the  hand  Is  positioned  over  the  cube  wjth  maximum  opening 
between  the  fingers  (2,4  Inch#*  between  the  tips  of  the 
toyoh  s*n8o.,s)  and  then  olosid#  the  cub?  will  be  muvsd  and 
ulwayj  end  In  the  same  position  relative  ,o  the  hand, 
Independent  of  the  Initial  position,  (See  Figure  30),  This 
motion  Is  nometlmes  disturbing  (e.g  when  grasping  the  too 
cube  of  a  stack#  the  movement  can  cause  it  to  become 
unstable  before  It  Is  fully  gripped  and  It  will  fall  off  the 
stack)#  and  henoe  no  use  Is  made  of  this  feature,  instead  we 
also  eorrect  errors  In  this  dilation  so  that  when  a  cube  Is 
grasped#  It  js  being  moved  from.  Its  original  looatlon  by 
less  than  the  tolerance  of  the  feedbaok  loop, 

The  grasping  Is  done  In  the  following  steps  (The  numbers  In 
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each  step  r®fer  to  figures): 

(a)  The  ringers  are  fully  opened  (by  UNGRASPing)  and 
and  the  hand  l8  MOVEdTO  over  the  center  of  the  cube  30  that 
the  fingers  are  parallel  to  the  cube's  sides.  The  cube  has 
at  most  two  vertical  faces  vlslple  to  the  camera.  One  of 
them  is  mor©  visible  to  the  camera  In  the  sense  that  Its 
normal  Is  pointing  closer  to  the  direction  of  the  camera 
center  than  the  normal  of  the  other  one,  We  call  this  face 
MWF,  (most  visible  vertical  faoe},  the  other  Is  oalied  SWF 
(second  visible  vertical  faoe),  The  hand  Is  oriented  so  that 
the  fingers  are  parallel  to  the  MWF ,  Th  I  s  wl||  help  to 
locate  the  hand-mark  In  the  image  more  precisely,  (31(a)) 
and  32(111)), 

(b)  The  touch  sensors  are  enabled  and  the  hand  is 
closed  slowly  (at  about  1/4  of. the  usual  speed  or  about  1 
Inch/sec)  till  one  of  the  fingers  touches  the  face  of  the 
cube,  As  mentioned  before#  the  touch  Is  light  enough  so 
that  the  Cube  Is  not  moved,  The  touch  sensors  are  then 
disabled,  (31(b)  and  32(IV)i, 

(o’  Using  the  following  Information  (for  which  wq 
assume  that  errors  can  be  neglected):  The  fingers  are 
parallel  to  the  cube's  face;  The  dlstanoe  between  the  tips 
of  the  sensors  after  the  closing  motion  of  the  fingers  is 
stooped;  The  thickness  of  the  fingers  between  the  tip  of  the 
sensor  and  the  hand-mark,  the  equation  for  the  plane 
containing  the  hand-mark  facing  the  camera  Is  computed  in 
the  t,c,s. 


(d)  The  hand-mark  Is  sought  using  FIND.HAND,  After 
the  two  corners  of  the  hand-mark  have  been  found,  the  camera 
transf ormatl on  Is  used  to  pompute  the  corresponding  rays  in 
c,t,c.  These  rays  are  intersected  with  the  plane  found  in 
step  (c)  to  give  the  c.t,e,  of  the  corners,  To  verify  that 
the  corners  found  do  belong  to  the  hand-mark,  w®  check  that 
they  have  approximately  the  same  height,  and  that  the 
distance  between  them  corresponds  to  the  width  of  the 
hand-mark . 

(e)  Usin9  the  Information  already  used  in  step  (c), 
the  position  errors  of  the  hand  are  computed,  If  the 
magnitude  of  the  errors  In  a||  three  directions  are  less 
than  a  threshold  (currently  ,1  Inch), 1  the  task  Is  finishea, 
we  go  to  step  (f)  and  then  exit.  If  the  errors  are 
larger,  the  hand  is  opened  < 3 2 ( V ) )  and  the  errors  are 
corrected  by  changing  the  position  of  the  arm  aoproor  late  I y . 
(31(e)  and  32 ( V I ) >  s  We  then  go  back  to  step  (b)  to  check 
errors  again,  (32(VII)). 
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Fig.  31. 
GRASPING  TASK 
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5.3  Placing  t®sk 


at  a  given 


„  th.  *..a  , .  2- ,0 ..e.  ' KSY* . it 

„c.ti°n  on  t«.  tMl.^  «lth  eur,  on  tny 

of  known  height  which  does  not  have  any 
location  where  the  cube  Is  to  oe 
support  hypothesis  Is  the  only 
The  task  is  carried  out  in  the 


could  he 

horizontal 

reference 

placed, 

externa  I 

following 


surface 
marks  near  the 
In  this  case  the 
Information  used, 
stops* 


Cl  Th.  .cub..J?._0R»|P.heu(33<(n).oi-nde'10VEdT0d. 


position  above  the  table  whare 

(33<IV>),  .  nnonnu,, 

(b)  The  cube  Is  PUTDOWN. 

(c)  The  hand  mark  Is 


( 33 ( V  )  ) , 
located 


In  the  Image  using 


FIND. HAND. 

( d ) 


The  camepe 


Is  centered  0n  the  vlslb|e  bottom 


,dge,  «»  th.  tjjhe;oo,t|ons  n|d-oolnt.  and  th.  orientation 
of  tht  image*  0f  the  two  visible  bottom  edges  of  the  cube 
are  comouted  using  the  hand  trasf ormat I  on,  the  information 
l!i%H«  JSS  «!•  th.  cub.,  Th.  .r.dlct.d  loe.tl."  <• 

th.n  c-r.ct.d^^th^.jount.  th,  two 

lines  In  the  Image.  The  two  lines  are  Intersected  to  fmd 
rnlr  location  in  the  Image.  Using  the  support 
^he*rhl«ia  the  t  t  c  iptatton  o f  the  corner  :*  computed 

rSHfc® 

il.I.Il.n, th*  f..  I.  Oth.r.!..  th.  cud.  I. 

back  to  step  <b>  to  check  the  errors  again.  C 33 c V 1 1  * ) ) . 
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5,4  StaoklnQ  task 

Tha  task  Is  to  stack  on*  cube  on  too  of  another  oube 
so  th»t  the  edges  of  the  bottom  face  of  the  too  cube  (BFTC) 
hl|l  be  biral|el  to  the  edges  of  the  too  face  of  the  bottom 
cube  (TFBC),  at  offsets  specified  to  the  program  by  the  ustr 
or  the  calling  module,  The  dlstanoes  are  specified 
perpend  I ou I ar  to  the  MVVF  and  SWF, 

The  tes*  Is  carried  out  In  the  following  steps? 

(a)  The  too  cube  Is  GRASPed,  (35(111)). 

(b)  The  camera  Is  oenterd  on  the  TFBC.  The 
mid-points  and  orientations  of  the  Images  of  the  two  edges 
of  the  TFBC#  belonging  also  to  the  MVVF  and  SWF.  are 
computed,  The  oorner-f I nder  Is  used  to  locate  these  two 
lines  In  the  Image,  The  locations  and  orientations  found 
are  then  stored.  The  two  lines  are  Intersected  and  using 
the  known  height  of  the  cube,  the  o.t.c,  of  the  location  of 
the  oorner  afe  found,  Using  the  given  offsets#  the  c.t.c,  of 
the  reaulred  positions  of  the  corner  of  the  BFTC  and  the 
center  of  the  top  cube  are  calculated.  (34(b)). 

<c)  The  too  cube  Is  MOVEdTO  a  location  just  above 
the  bottom  cube,  oriented  so  that  the  hand-mark  Is  parallel 
to  MVVF  of  the  bottom  cube,  The  h.t.c.  of  the  bottom  cube 
(WM0)  a  re  taken  Into  acoount  If  known.  (35(IV)), 

(d)  The  top  cub*  Is  PUTDOWN  on  the  bottom  cube, 
( 34 ( o )  to  ( e )  and  35 ( V  )  )  , 

(e)  The  hand-mark  Is  located  In  the  image  and  then 
the  two  edges  of  the  BFTC  as  done  In  steps  (o>  to  (f>  of  the 
placing  task,  In  this  case,  however*  the  edges  of  the  TFBC 
will  also  aDpaar  In  v|#w,  A  slmo|e  algorithm  is  used  with 
the  Information  computed  In  st*D  (b)  to  decide  which  or  the 
lines  are  the  edges  of  the  BFTC.  mis  simple  algorithm  can 
be  deceived  sometimes  by  the  presence  of  shadows  end 
"doubling"  0f  edges  In  the  Image,  We  could  make  the 
algorithm  more  Immune  by  locating  the  vertical  edges  of 
cubes  also  If  they  could  be  found, 

(f)  The  two  edges  of  tb®  BFTC  found  In  the  last  steo 
are  Intersected  to  find  the  corner  location  In  the  Image, 
Using  the  sUooort  hypothesis,  the  c.t.o.  of  the  location  of 
the  corner  Is  computed  end  compared  with  the  reaulred 
location  oomouted  In  step  (b),  j*  the  magnitudes  of  the 
errors  are  less  then  a  thresho'd  (.1  Inch)  in  both 
directions  then  the  task  Is  comoleteo,  Otherwise  the  cube  is 
LIFTed  ( 35 ( V I ) >  and  the  error  corrected  by  changing  the 
position  of  the  arm  app r oo r I  ate  I y .  (34(f)  and  35 ( V  S  X ) ) •  Then 
we  g0  back  to  step  (d)  t0  oheck  the  errors  aga!n,  (34(d) 
again  and  35(VIII)), 
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Instead  of  ■  bottom  cube.  «e  scecl'y  to  the 
program  •  square  hoi*  In  *  bottom  ob J*ot  In  whloh  the  too 
cub#  Is  to  be  |nsert*fli  [«  this  a**er  when  the  too  oub* 
placed  on  the  bottom  object  "•  to  oheoK  ho»  much  It  mas 

lowered.  If  It  w*e  lowered  oeit  eoM  threshold  It  meam 
that  It  is  already  In  the  no|#  and  can  b*  raleit-e.  vie  niv» 
to  make  sur*  that  the  orlp  of  the  ht^d  l»  tight  enough  eo 
that  the  eube  graeped  will  not  rotate  whOn  Ofacie  pertly 
above  the  hole, 

This  modification  was  Implemented  and  successfully 
tried  by  R,  Oavls  of  tha  hand-eye  group  for  other  objects 
(wedge  and  slab)  also.  (See  Flour*  36), 
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chapter  6 i  summary  and  future  work 

6,1  Introduction 

The  actual  results  and  performance  evaluation  of  the 
thrae  major  components  of  this  researoh,  (corner-f Inder, 
oa|lbratlon  updating  and  visual  feedback),  are  discussed  In 
the  next  three  seotlons,  Slnoe  the  performance  of  latter 
components  re|lea  on  the  performance  of  prior  ones,  these 
sections  should  be  read  In  ordar,  With  the  perf  orme.-ice 
desorlbtlon  w»  dlsouss  the  major  shortcomings  of  the  three 
components  and  their  effeots,  and  suggest  possible  future 
Improvements  or  a ' ternat I ves ,  Some  of  these  have  already 
been  dlsoussed  In  past  chaptersi  In  the  last  seotlon  wo 
dlsouss  possible  directions  for  further  research,  to  extend 
and  generalize  the  results  accomplished  In  this  work, 

Much  effort  went  Into  making  the  programs  effective 
and  reliable,  Eaoh  program  can  have,  In  general,  five  kinds 
of  results:  <a)  It  aohfeved  the  goal  set  to  Itj  (b)  It  quit 
trying  for  justifiable  reasons  (In  a  sense,  It  took  account 
of  Its  own  limitations))  ( c )  It  Quit  trying  whan  there  was 
no  reason  to  do  so  (If  It  was  a  person  we  would  say  that  he 
was  lazy  or  not  persistent  enough);  ( d >  It  achieved  the 
wrong  goal  but  realized  It  (In  this  case  there  Is  some 
reason  not  to  simply  try  again))  (e>  It  achieved  the  wrong 
goal  but  did  not  realize  It,  In  the  last  two  cases  we  could 
also  have  undesirable  side-effects  like  undoing  a  goal 
already  achieved,  loss  of  Information,  or  even  hardware 
damage , 


By  '‘effective"  we  mean  that  for  the  restrictive 
conditions  under  which  wp  operate,  the  percentage  of  type 
(a)  results  will  be  high,  By  "reliable"  we  mean  that  the 
percentage  of  tyo*s  (d)  and  (e)  results  will  be  very  low 
and,  when  the  side-effects  Inolude  hardware  damage, 
extremely  low. 

Each  program  was  run  several  hundred  times  and  In 
the  case  of  the  corner* f I nder  several  thousand,  No  formal 
statistics  were  gathered,  for  reasons  that  are  discussed 
below,  but  estimated  numbers  for  tho  effectiveness  and 
reliability  will  be  g I  van, 

There  are  two  general  types  of  causes  for  not 
aohlevlng  the  r®su!ts,  after  all  the  programming  errors, 
user  errors  or  hardware  failures  (Section  7,2,2)  are  either 
eliminated  or  discounted,  One  Is  exoesslve  noise  In  the 
measurements  of  either  the  Intensities  of  the  Image  or 
various  parameters  of  the  system,  The  other  Is  tne 
partioular  details  of  a  given  situation,  such  as  orientation 
of  the  objeots,  their  colors,  their  relative  position, 
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I  looting  conditions#  etc,  which  make  the  task  difficult. 
Th*  effects  of  both  causes  ap«  usually  compounded, 

The  complexity  of  the  situation  and  of  the  system 
makes  It  extremely  difficult  to  analyze  It  theoretically#  Qr 
to  make  statistically  controlled  experiments  (of  the 
Monte  Carlo  type  for  example),  Therefore  the  only  kind  of 
numbers  that  we  will  cite  are  estimated  from  long  exDerlance 
with  the  system,  Hopefully  they  are  not  biased  by  our 
wishes#  or  by  the  effect  of  unconsciously  "doctoring"  the 
conditions  under  which  the  experiments  are  executed,  since, 
by  properly  (really  Improperly)  choosing  the  conditions#  a 
very  high  percentage  of  success  could  be  achieved, 

6,2  Corner-f j ndep  performance 


While  operating  within  the  assumptions  mentioned  in 
Section  0,1,3#  the  effectlvness  of  the  corner-f fnder  is 
estimated  at  more  than  90*,  When  form  and  intensity  matohe#? 
are  demanded  the  effectlvness  Is  even  higher. 

The  Program  needs  less  than  ,5  seconds  to  analyze 
the  regu|ar  18*18  raster  nit  window  (without  r scan te r I  no ) , 
Another  .5  seconds  Is  needed  to  display  the  Information 
(intensity  mao,  boundary  points#  fitted  lines  and  features 
data  structure)  for  the  user  when  requested. 

There  are  two  situations  which  occur  In  the  stacking 
task  when  some  of  the  assumptions  are  violated  and  the 
performance  is  degraded,  The  first  happens  when  two  adjacent 
faces  have  very  similar  Intensity,  The  other  happens  when 
the  Images  of  two  faces  are  separated  by  a  narrow  strio  <2 
to  3  raster  units  wide)  with  different  but  somewhat 
widespread  Intensities,  In  both  situations#  sometimes#  the 
histogram  does  not  have  the  proper  number  of  maxima, 


The  first  situation  Is  a  tough  nut  for  any  scene 
analyzer,  T0  find  the  boundary  In  this  oase,  fine  tuning  of 
the  sensitivity  of  the  camera#  and  some  kind  of  statistical 
analysis  is  needed,  For  example  the  line-verifier  described 
by  Tenenbaum  C28],  The  solution  will  be  much  simpler  If  the 
Intensities  are  similar  when  viewed  through  the  clear  filter 
but  actually  the  faoes  have  different  colors,  If  the  colors 
•re  known. to  the  Program  It  can  select  ths  filter  that  will 
maximize  contrast,  If  the  colors  are  not  known  but 
difference  of  colors  Is  suspected,  the  program  can  try  tne 
filters  In  turn  till  one  (|f  any)  gives  enough  contrast, 


We  tried  to  avpld  the  first  situation  i>y  carefully 
painting  the  faces  of  the  cubes  In  different  shades  of  dray, 
Different  magnitudes  and  directions  of  errors  can  cause 
different  faces  to  meet,  We  need  at  least  throe  o'jffemt 
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I nt*n« I ties,  Two  of  them  (of  the  vertical  faces)  have  to  be 
different  from  the  background  Intensity  also,  (See  Figure 
37) , 

Thpee  Uternatlvas  are  suggested  to  solve  the 
problem  of  the  seoond  situation! 

(a)  A  modified  version  of  the  corner-finder  with 
hlst0Oram  ana  |  y  s 1 s  ta I  1 o  r • d  tc  the  presence  0f  narrow 
s  t  r !  0  8 1 

(b)  An  Improved  version  of  the  operator  desorlbed  by 
Hueckel  C 12 3  that  can  detect  narrow  strips  also,  which  is 
now  being  Integrated  Into  the  hand-eye  system, 

(c)  Us«  of  a  higher  magnification  lens  which  will 
cause  the  Image  of  the  strip  to  be  wide  enough  for  the 
corner-finder.  The  disadvantage  of  this  solution  Is  that 
more  effort  wl I  I  be  needed  for  oenterlng  and  for  focussing 
the  camera,  beoaus*  now  the  f I e  ld-of-v I ew  and  depth-of -f  I  e  I  d 
are  sma I | er , 


6.3  Calibration  updating  programs  performance, 

6.3,1  Pan/t I  1 1  calibration  updating 

When  the  degradation  of  coordination  causes  errors 
of  lees  than  30  raster  units  on  the  Image,  or  about  1  Inch 
at  typical  operating  rangt»  the  pan/tllt  calibration 
updating  from  the  arm  will  seldom  fall, 

The  major  reason  for  the  effectiveness  Is  that  we  can 
oontrol  most  of  the  factors  of  the  soene! 

(a)  The  environment  Is  simple  and  hence  we  can 

searoh, 

(b)  The  hand  Is  rotated  so  that  the  hand-mark  will 
face  the  camera  obsourlng  othep  features  of  the  hand  that 
are  more  complicated, 

(c)  The  finger  and  hand-mark  were  painted  to  give 
high  resolution, 

(d)  The  dimensions  of  the  hand-mark  were  chosen 
(constrained  by  the  finger's  s|ze)  to  be  oompatlbte  with  the 
-egular  18*18  raster  units  window  and  the  2  Inch  lens.  The 
Image  of  the  hand  mark  Is  bigger  than  the  window  but  the 
Image  of  the  f|nger  beside  the  hand-mark  Is  not  too  narrow, 

(e)  The  form  of  the  hand-mark  (rectangle)  was  chosen 
so  that  the  simple  eearoh  algorithm  already  built  Into  the 
oorncr-f I nder  will  lead  to  the  right-hand  lower  corner 
whenever  any  part  of  the  hand-mark  Is  seen,  (The  reader  oan 
simulate  this  action,  similar  to  the  manner  emoloyed  in 
Figure  23),  Jf  needed,  the  form  of  the  hand-mark  could  be 
made  more  complex  to  avoid  confusion  with  other  objeots, 


(a)  Ar rangement  of  minimum  number  of  intensities. 

(b)  Arrangement  of  preferred  number  of  intensities. 


Fig-  37- 

PAINTING  SCHEMES  FOR  VISUAL  FEEDBACK  TASKS  FOR  USE 
WITH  THE  CORNER-FINDER 


In  this  case  also,  we  oan  assure  high  effectlvness  by 
employing  a  more  complex  searob  algorithm  tuned  to  the  known 
form  of  the  hand-mark, 


.  .  sj.hCe  the  relative  Intensity  and  form  bre 
a  form  end  Intensity  matoh  is  obviously  demanded, 


known, 


IS8  !ir!*  !rmH?09iflon  UMd  15  th#  f*r the»t  from  the 
camera  so  that  coordination  errors  even  larger  than  1  inch 

can  ba  tolerated,  if  toe  program  succeeds  in  locating  the 
hand-mark  In  the  first  arm  position,  the  "Instant 
adaptation"  described  In  Section  4,4,  and  the  fact  that  a 
large  part  of  the  error  Is  deterministic  In  nature,  win 
Insure  the  Program  suocess  at  suaseauent  positions  of  tne 
arm, 


In  a  time 
coordination  can 
Image  of  up  to  20 
Figure  27),  and 


than 

6  to 

less 

than  3 

needed 

for 

one 

run 

( 13 

25  minutes, 

More 

than 

ha  |  f 

Interval  of  two  weeks,  degradation  of 
cause  averaged  prediction  error  in  the 
raster  units  in  the  x-dlrectlon  (see 
_  ,  ,  -  10  raster  units  In  the  y-d|rectlon. 

Occasionally  (orobably  because  of  unauthorized  tinkering), 

t  ,  iwapr?rs  ir*  |ar0,,r*  °ne,  or  at  most  two  runs  of  the 

calibration  updating  program  w|||  reduce  the  averaged  error 
to  less  than  4  raster  units.  The  standard  deviation  of  the 
errors  w|||  be  reduoed  from  less 
raster  units, 

The  CpU  processing  time 
positions  of  the  arm)  |3  about  1 

of  this  time  |s  used  for  the  real  time  control  of  the 
camera,  A  new  camera,  now  being  added  to  the  hand-eye 
system,  will  move  5  times  fastar, 

pa i ihratlnn  ,e, f l 8 2? ' V 8 n S 9 3  and  Processing  time  naeded  for 
calibration  updating  from  cubes  on  the  table  tio  not  make 

nruch  sense  Since  most  of  the  w0rk  Is  done  by  the  U3er,  and 
hence  will  n0t  b9  cited, 

6,3,2  Focus  calibration  updating 

„  Un|ika  Pan  *nd  tllti  degradation  In  focus  Is  rarely 

noticed  and  the  program  Is  run  only  after  the  camera  is 
taken  apart  for  repair  or  ma I nta I  nance , 

.  Using  the  automatic  focussing  routine  is  very  time 
consuming  because  the  foous  Is  servoed  20  to  40  times  oap 
range  point  (w8  use  8  different  ranges),  Therefore  the 
manual  mode  Is  used  most  of  th#  time, 


6,4  Visual  feedback  tasks  performance 


From  long  experience  with  the 
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system,  the 


effectiveness  o'  the  grasping  ana  placing  tasks  is  estimated 
to  be  ab0v,  85%#  while  that  of  the  stacking  task  is 
estimated  ta  be  about  70%.  Th#  failures  of  the  stacking 
tasks  are  caused  maln|y  by  failures  of  the  co^ner-f :rder . 
Reasons  and  possible  Improvements  have  already  been 
discussed  In  Sect  I  on  6.2,  y 


The  second  cause  of  failure  in  the  stacking  tisk  i  •> 
rovement  o'  the  bottom  cube  occurring  when  the  toe  cube  |5 
placed  on  It.  Jn  similar  cases  we  (persons)  will  -se  tn« 
otner  hand  to  hold  the  bottom  cube  so  that  It  will  net  move 
a  similar  solution  snould  be  Investigated  when  the  hand-eye 
system  "grows"  another  arm,  Till  then  (some  months  from 
row),  we  t r y  to  Prevent  movement  by  using  heavy  cubes, 
(which  are  needed  also  for  proper  functioning  of  tne  touch 
sensors)#  and  •  black  rubber  pad  to  cover  the  table,  As  an 
elternatlve  we  c*n  try  to  locate  the  bottom  cube  again  each 
time  after  the  too  cube  was  placed  cn  It.  If  enough  edges 
and/or  corn8fs  of  the  bottom  cuoe  are  not  obscured 


The  intricacies  of  the  world  mode |  and  Its  upoatin 
help  to  maintain  the  reliability  at  a  very  hlgn  level, 


The  coordination  errors  corrected  by  the  visual 
feedoack  have  mognltude  uo  to  .6  inches,  j  f  tnka  into 
account  the  maximum  opening  between  the  M^ge-s  (2, 4 
Inches),  the ;  r  wloth  (.75  Inches)  and  the  size  of  the  cubes 
handled  (1,25  and  1,50  inches)#  it  can  be  seen  t n  a  t  t n #. 
magnitude  0f  the  error  corrected  ls  dose  to  the  magnitude 
of  the  errors  jhat  can  be  tolerated  before  the  nature  of  tnp 
situation  changes,  especially  |n  the  grasping  task. 


If  the  magnitude  of  the  coordination  errors  is  vor 
larQe,  the  ‘'and  can  Press  the  cube  frrm  above  «h‘io  tr>ln.i 
to  get  ever  it  (see  Figure  38(a)),  or  the  t>uch  sensors,  or 
even  the  whole  fingers  can  miss  the  cube  when  closing,  "ncs* 
situations  aPe  not  handleo  currently,  (See  Figure  33(c)  p.n~ 
(b) ), 


..  .  Th«  first  situation  can  be  detected  and  corrected  ny 

finding  the  c.t.c,  of  th#  arm  after  It  stooped  becaus®  of 
thf  opposing  force.  The  support  hypothesis  Is  used  *  3 
external  Information,  Un||ke  Placing,  the  pressure  jn  t”  - 
case  Is  less  distributed  anp  can  cause  movement  of  the 
Because  0f  the  *b0ve  reason  wc  did  not  handl#  th I s  case"  ano 
decided  to  wa I t  for  more  sensitive  force  sensing,  -r  mere 
touch  sensor,  (See  Figure  38(d)), 

The  second  situation  can  be  detected  as  'oi|CW~:  t 

the  coordination  errors  are  very'larSe#  the  f inq0rs  wiii 
miss  the  cube  completely  and  close  till  they  teuen  eac* 
other  and  then  stop  oecause  a  touch  sensor  was  activated. 


t 


« 


f 


« 
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The  opening  between  the  fingers  In  this  case  u  zero  (see 
Figure  38(b)),  If  the  orrors  ar*  of  lesser  magnitude,  the 
touch  sensors  will  miss  the  cube  but  the  rubber  pads  on  the 
fingers  will  grl0  U  (see  Figure  38(c)),  In  this  case  the 
ooenlng  between  the  fingers  will  be  almost  the  size  of  the 
cub*  but  no  touoh  will  be  Indicated, 

Finding  the  coordination  error  In  the  second 
situation  Is  more  difficult  than  In  the  first  beoause  we  do 
not  have  any  external  Information  to  use.  In  the  *lrst  ease 
we  can  find  the  direction  of  the  error  by  observing  if  the 
cube  obscures  the  hand  or  vice-versa,  and  correct  by  a 
cut-ard-try  method,  The  second  case  Is  even  worse  because 
the  c.t.c,  of  the  cube  are  lost  by  the  sweeping  motion  of 
the  fingers,  Th|s  problem  can  also  be  solved  oy  adding  more 
touch  sensors  so  that  the  fingers  cannot  touch  anything 
without  at  least  one  touoh  sensor  being  activated  (see 
Figure  38(d)), 

Similar  situations  can  ocour  In  tn#  stacking  task, 

Currently  the  Incremental  movement,  of  the  am  is 
realized  by  changing  the  angles  of  all  six  joints  of  the 

I".  th.  accuracy  o«  th . .  U  ""  f.d  by  th. 

precision  of  measuring  tha  ooslvlon  of  the  first  three 
Joints  (  tha  “shoulder"  Joints).  A  design  of  a  new  hand  is 
now  In  progress  that  wll|  enable  us  to  r ea  I 1 ze  . I nc r •njnta i| 
motion  of  tha  hand  In  two  peroand I ou  |  ar  directions  without 
moving  the  arm  at  all,  It  will  oa  accomollshad  by  connecting 
the  hand  to  the  am  with  an  axtendlb|a  rod  and  by  separate 
control  of  aach  finger.  Tha  positional  errors  In  this  cose 
will  be  much  less  than  bafore  since  tha  same  relative  error 
Is  now  multiplied  by  a  much  shorter  length, 

t ,5  Suggestion  for  future  research 

Each  0f  tha  three  research  components  suoo*sts 
olrectlons  for  future  research  to  extend  and  generalize  the 
results  reoOrted  her#! 

(a)  Corner-f lnder I  Othir  scene  analysis  or  I nc i D I  as* 
(gradient  following,  region  growing!  use  of  color  and 
texture,  direct  three  dimensional  measurement.  ®tc.)  should 
be  adapted  for  u$e  In  visual  feedbaok  and  sm.lar 
applications,  (I. a  the  programs  using  these  or'nclples 
should  be  able  to  use  already  known  Information  at  all 
levels  of  Its  ooeratlon), 

(b)  Calibration  uodatlngi  The  updating  ?r0J;?*9 
should  ba  extended  to  more  oarameters,  For  example!  A  white 
hand-mark  c0u|d  halo  to  update  the  color  calibration  of  the 
system  wh9n  the  lighting  conditions  ar®  changing, 
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Calibration  updating  using  arrors  detected  in  regular 
operation  of  the  system  should  be  Investigated  Including 
orobleme  of  utl'lzlng  errors  datactad  with  low  confidence. 
Calibration  uodatlng  of  tha  arm  can  also  be  tried, 
especially  after  tha  systam  wt|l  have  two  cameras  wnicn  can, 
In  principle,  define  thr«a  dimensional  coordinates  without 
using  the  table#  or  tha  suooort  hypothesis  In  general,  Tnn 
problem  of  cor r asoondanoa  Is  relatively  easy  to  solve  when 
only  the  h*nd  Is  In  view, 

(c)  visual  feedbaok  tasks:  The  capabilities  should 
oa  extended  to  nandie  more  types  of  objects  and  relations 
between  them.  This  means  not  Just  writing  more  orograms  to 
handle  more  situations,  but  rather  generalizing  the 
principles  of  ooaratlon  to  Include  mere  typas  of  ooJect3  and 
relations, 

We  can  now  describe  what  might  be  the  next  steps  of 
the  development  of  the  stacking  task: 

(a)  a  pair  of  edges  are  specifies  on  the  *cp  'ace  o ( 
the  bottom  ofcject  end  another  pair  on  the  oottom  'ace  of  the 
too  object  to  b«  mated  to  the  first  pa  I r  or  to  satisfy  some 
given  relation.  All  four  eodss  have  to  oe  vlsiole. 

<b)  The  too  and  bottom  facos  aro  specified  with  tne 
relation  they  have  to  satisfy,  In  this  case  the  program  has 
to  choose  edges  and  relations  to  check. 

The  programs  to  accomplish  the  generalized  task  will 
need  heuristics  for  automatic  planning  of  '.he  sco-e  analysis 
reeded  for  various  manipulated  objects  and  relations,  Tnp 
oiannlng  w|||  have  to  take  Into  account  the  fact  fat  whew 
tne  ooject  to  b«  moved  can  be  grasped  In  more  than  one  *ay 
(or  lantatlonj ,  the  orogram  can  choosa  the  way  which  wju 
rake  the  scene  analysis  aaslast,  Currently  the  scene 
analysis  and  selection  are  written  explicitly  Into  the 
program  for  cubes, 

In  parallel,  fa  subject  of  tactile  feedback  should 
be  davelooed  to  use  In  situations  where  the  view  pf  the 
cemere(s)  is  blocked,  and  to  augment  the  visual  feeboaek, 

Some  examoie$  of  oossibla  augmen-at i on  were  giv^n  I-  tm 
last  section. 

In  t^e  next  (end  last)  chapter  *e  citcuss  s  "•-? 
tareral  orotjlems  that  In  mv  colnion  are  toe  important  tr 
gioss  over  o*  to  cut  in  an  apoeno'X, 


CHAPTER  7?  "GENERAl*'  PROBLEMS 
7,1  I ntroduct I  on 

The  ouroose  of  this  chaotor  Is  to  bring  to  light  a 
number  of  Problems  related  to  the  design  of  robots  or  othor 
lerfl*  scale  systems,  Most  of  the  oroblems  are  not  now,  They 
•re  brought  uo  here  because  examples  of  them  surfaced  during 
the  research,  These  examples  ar«  used  to  exolein  the  nature 
of  the  general  Problems,  For  some  of  them  a  genorai 
principle  of  so'utlon  is  suggested  with  a  3hort  description 
of  the  particular  ways  these  solutions  were  Implemented,  For 
some  of  them  no  satisfactory  solutions  were  found, 

The  problem  and  solution  evoiveo  In  the  following 

feshlon: 


particular  oroblam  -■»  general  problem  --  general 
principle  Of  solution  --  particular  implementation 
of  solution 

Not  a | I  the  problems  aPe  of  the  samo  Importance  as 

the  reader  can  easily  s*e. 


7,2  Error  sources  and  handling 
7,2,1  General  not®S 

The  following  charact*r 1 st I cs  of  our  system  make  the 
problem  of  error  and  failure  handling  both  acuto  and  of 
general  lnterests 

(a)  The  programs  have  control  of  external  hardware 
(camera#  arm,  etc.).  The  real  time  nature  of  the  control  is 
sometimes  cruclal  ('or  example#  the  arm  control  takes  tne 
dynamics  of  the  arm  Into  account),  The  hardware  does  not 
have  anouflh  safeguards  of  Its  own  because  of  oesljn 
oec  I  s I ons  #  and  not  because  of  neglect.  For  example,  tnore  i3 
no  simple  way  to  Install  mechanical  stops  or  microswltc  es 
on  the  arm  to  prevent  It  from  striking  the  camera, 

(b)  The  collection  of  programs  Is  very  big  {  several 
hundred  thousand  words)  and  written  by  different  people  3t 
different  tlmes, 

(c)  The  main  processor  and  some  of  the  other  devices 
are  time  shared  between  a  number  of  unrelated  Jobs,  Thjs 
character Istic  will  still  hold  if  the  feoillties  are 
dedicated  but  the  orflan  zatlon  of  thf  orograms  does  not 
Inolude  an  Omniscient  central  controller. 

In  the  following  sections  wo  w|||  i|5t  possible 
error  sources#  the  effects  of  errors  end  some  Problems  eng 
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solutions  concerning  the  handling  of  errors  ana  fa!  j^pj. 

7,2,2  Error  sources 

Tne  following  in  son*  of  th*  common  *rr or  sources! 

(a)  Programmer  errors*  This  Is  the  most  usual  kind, 
where  th*  prod'emmer  mistyped  or  used  the  wrong  eauatlon, 
This  kind  of  error  Is  common  during  the  development  chase 
and  hopefully  eliminated  from  the  experimental  and 
operational  phases, 

(b)  Program  errorsi  Programs  of  course  do  not 
oenerate  errors  of  th* j r  own,  so  that  this  kind  of  error  can 
also  be  attributed  to  the  programmer,  They  ar*  more  suatle 
end  less  explicit  than  thos*  of  kind  (a),  They  occur 
when  the  complexity  of  the  program  caures  a  situation  which 
the  programmer  did  not  think  oosstble  or  did  not  realize 
oould  exist  at  all,  and  so  did  not  make  the  right  provisions 
In  the  program  for  the  situation. 

(c)  Hardware  errors!  This  typ*  of  error  Is 
Inevitable,  The  frequency  of  their  occurrence  can  oe 
reduced  by  making  the  hardware  mora  reliable,  Elimination 
(l,a  making  the  probability  of  occurrence  extremely  low) 
requires  a  high  dagr*a  of  redundancy  (as  exists  for  example 
In  animals),  This  Is  too  expensive  In  most  oai>es,  Hardware 
errors  ar#  e«used  either  by  malfunction  or  by  noise  Cor  In 
general  by  son#  parameter  getting  out  of  Ita  soeclfled 
range ) • 


(d)  user  errors?  During  the  development  phase,  and 
aft*r#  th#  Programs  lnt*raot  with  the  us*r,  By  accident,  or 
ignorance  (or  malice)#  the  user  cen  feed  the  program 
Inoorract  Information, 

7.2.3  Effects  of  *r  rOrs 

Th#  effects  of  errors  rang#  from  task  IncgmDlatlon 
through  th#  antisocial  tying  up  of  the  processor,  to  the 
major  catastrophe  pf  hardware  destruction  or  even  human 
Injury,  We  would  lie  to  avoid  the  first  two  ana  completely 
eliminate  th#  third  If  possible, 

7.2.4  Error  detection 

The  following  oons I deret I ons  affect  the  design  of 
error  detection? 

(a)  To  Insure  against  hardware  malfunction  or 
destruction  w#  would  like  the  routines  which  aotually 
operate  the  hardware  (suoh  ae  the  arm  and  camera  servo 
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Droflrams>  to  check  all  the  oarameters  given  to  them, 

(b)  Leaving  the  error  checking  tUl  the  last 
Doeaible  moment  la  uneconomic  In  the  sense  that  we  would 
save  much  time  wasted  In  calculations  that  become  use  leas 
when  the  efror  Is  detected,  Hanoe  we  would  like  to  detect 
the  error  as  soon  after  Its  generation  as  possible,  However, 
the  determination  where  a  chain  of  computation  started  with 
a  parameter  that  eventually  wl||  generate  an  error  Is  very 
difficult. 


khuh  *2  t(?®  programs  described  here,  all  the  routines 

which  communicate  with  the  hardware  were  programmed  to 

check  the  parameters  given  to  them,  Detection  of  errors  at 
an  earlier  stage  was  left  optional, 

7,2,5  Errors  and  failures  handling 

Detected  errors  are  handled  like  failures.  Failure* 
are  situations  where  a  routine  cannot  comolete  Its  "ask  or 
function,  unlike  errors#  failures  happen  naturally  In  any 
pr°gram  that  uses  search  or  cut-and-try  methods,  Sometimes 
the  failure  of  a  subroutine  means  success  of  the  railing 
routine  If  |t  wanted  to  refute  a  hypothesis  chat  the 

subroutine  was  a8ked  to  check,  For  exampler  If  we  want  to 
prove  that  a  given  space  on  the  table  top  Is  empty,  we  call 
the  corner-f | nder  and  ask  It  to  find  any  feature  there,  If 
the  subroutine  SLICE  (see  Section  3.2,2.)  reports  a  failure 
(l.e  it  found  that  the  Intensity  In  the  window  ls  uniform  or 
that  the  Intensity  histogram  has  only  one  maximum),  then  we 
have  succeeded  In  proving  t<»at  the  space  was  Indaed  empty, 

Error  and  failure  handling  poses  the  followlno 
general  problem  that  I  w|||  try  to  explain  with  an  example? 

L**  U3  .Say  that  W®  hav®  *  r  0IJt  I  ne  ri  that  can  terminate 
either  with  sucoess  or  with  failure,  Nr#xt  w*  design  a 

routine  R2  which  calls  Rl.  After  Rl  Is  called  by  R?  and 

executed,  R2  checks  the  termination  condition  of  Rl,  If  Ri 
fal  ad,  R2  w|||  then  execute  some  action  A2,  Later  we  design 
a  routine  R3  which  also  calls  Rl,  Upon  failure  of  Rl,  R3 
executes  A3,  So  far  so  good#  put  now  comes  the  trouble.  We 
"*",  “■**  t*(i  mor*  routi#!*!  m  Rs  ,*  h  0,  lh 

«'  <"■  ***  w^ujd  ll-t  R?  ca  rrv  gut  act^m  If 

Rl  ■  Isi  d  [|k*  *□  carry  put  action  ;f  di 

f  i*  fro*  each  other  awg  both 

fflghi  m  different  1 rom  |h#  origin*;  ij) .  Jf  -estep 

eel  ling  would  stop  hore  we  night  go  back  end  change  R^so 
that  It  would  know  whether  It  was  called  by  R4,  or  by  R5,  or 
by  neither#  and  carry  out  the  appropriate  action,  Put  the 
design  Process  goes  on,  and  on,  and  R6,  R7  are  designed  each 
calling  R2,  R3,  Rd»  R5  at  various  stages  and  so  on. 
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An  alternative  solution  is  to  assign  a  suoer-Urnr 
error  and  failure  handling  routine  which  will  b«  called  ;  h 
tlm*  an  error  Is  dat*ct*d  or  a  failure  Is  Indicatoo  I r  r  o - 
now  on  we  will  refer  on|y  to  failures  but  vie  will  naan 
both).  Tnis  routine  will  have  access  to  the  nested  calls 
stack  and  1 n  general  to  the  state  of  the  process.  It  would 
then  fix  the  state  as  bast  It  could  and  transfer  control 
aopr od r  I  a te I y ,  In  simple  ces*s  It  would  return  central  to 
the  lnterruo?ed  routine, 

The  advantage  of  this  scheme  Is  that  n-rors  an" 
failures  are  dealt  with  on  the  spot,  Also  no  redesign  of  tv 
routines  Is  needed  each  tine  a  new  subroutine  ww  |  on  cn jsoc 
then  to  oe  called  Is  addeo,  The  disadvantage  is  conplexitv 
since  the  error  handling  routine  has  to  provic-  for  all 
contingencies,  rinding  alt  the  contingencies  Is  not  a 
trivial  oroblan,  Also.  tha  error  n^ncier  will  have  to  ot- 
changed,  or  at  least  addad  to*  aaeh  tine  a  new  routine  is 
designed,  me  transfer  of  control  needed,  and  tne  aoility  to 
Interrogate  the  full  state  of  tne  program*  cannot  be 
Implananted  In  SAIL  (and  shl  i|r  languages)  In  a  straight¬ 
forward  nanner,  It  oould  be  done  In  the  Improved  version  of 
SAIL  now  beln9  desloned. 

In  our  case  the  disadvantages  were  prohibitive  n' 
wa  useo  the  first  schema  (mentioned  at  the  eno  of  the  above 
exanpia),  without  nuch  redesign,  In  tne  terns  cf  the  last 
example,  we  *ould  not  redesign  R2  after  oesigrlrg  R4  and  r=. 
what  would  tnan  happan  Is  the  f  c  l  lowing}  failure  o'  D1  <n  =  ". 
called  by  R2  which  In  turn  was  celled  by  34  (or  Rf)  -00I1 
cause  R2  to  take  action  A2,  but  since  thjj  action  is 
I  naDor  opr  lata  It  would  oause  failure  cf  R2  also,  'new  R4  (c- 
R5)  can  deal  differently  with  the  failure,  alDelt  a  »  5 1 1 »  - 
lata  (It  cou|d  ba  worse  If  there  were  more  nested  cults 
levels),  The  Information  that  R1  Is  the  real  causs  of  tne 
faMu-e  can  o*  (  nd  Is)  filtered  uo  to  R4  ano  R5  Und  if 
needed  even  higher  in  the  hierarchy),  Note  that  leaving  v.p 
error  handling  to  higher  level  routines  has  the  aevant^se 
that  these  routines  w|||  have  more  general  knowledge,  ovt  cr 
the  othor  nano  this  knowledge  can  be  toe  condensec  and  not 
detailed  enough  to  handle  the  error  efficiently, 

There  Is  no  simple  solution  to  this  problem,  t  very 
complicated  answer  might  be  planning,  when  the  s>ste~  1- 
given  a  goal  It  will  plan  a  serios  of  suogo&ls,  cnecvs, 
branches  and  loops  leading  to  this  goal.  This  gross  D'a^nln- 
will  not  De  detailed  enough  to  bangle  all  orrors  rr  faii-raa 
and  hance  each  subg0*l  has  t0  b9  ana|yzed  and  rl?-nrp"  'or, 
This  should  be  receatad  for  lower  levels  till  we  reach  "-ho 
level  where  tne  built-in  failure  handling  I  <»  reliable 
enough,  This  o|annlng  for  errors  and  failures  Is  analogous 
to  the  automatic  writing,  specifically  for  «acn  goal  end 
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tailored  to  the  contingencies  anticipated,  of  a  suoer-duoer 
error  handilnfl  Program,  similar  to  the  one  mentioned  above, 

This  soheme  Is  somewhat  wasteful  and  we  would  like 
to  be  able  to  "store"  the  detailed  plans.  Slnoe  each  goal  Is 
at  least  a  little  different  from  preceding  goals,  a 
"generalized"  but  detailed  (see  the  contradiction?)  version 
of  the  plans  should  be  stored,  Then  we  will  use  an  execution 
monitor  that  w|||  fill  In  the  missing  details  on  the  go  and 
without  repeating  most  of  the  effort  that  went  Into 
planning,  Work  in  a  similar  direction  Is  In  progress  at 
S,R,I  C15 J , 


7,3  Empirical  parameters 

Many  of  the  programs  developed  through  this  work, 
especially  those  deajlng  with  Image  processing,  have  many 
parameters,  mostly  In  the  form  of  thresholds  which  ara 
empirically  fixed  at  a  oerttln  value,  During  the  program 
oeve I opmant »  no  clear  method  for  the  determination  of  their 
optimal  values*  for  various  anticipated  situations,  was 
found  that  could  than  be  given  to  a  program,  Hence  they  were 
fixed  at  some  valua  thought  to  be  optimal  for  most 
situations,  They  are  too  numerous  to  be  passed  as  formal 
parameters  and,  beoause  of  the  reason  stated  above,  too 
confusing  to  be  treated  ae  suofc,  On  the  other  hand  we  would 
like  to  make  them  visible  to  a  future  user  who  will  have 
situations  not  anticipated  at  design  time  which  need  either 
a  change  In  the  fixed  optimal  value  or  a  variable  value  for 
which  an  algorithm  oan  be  found,  No  Pood  solution  was  found 
to  this  Problem  apart  from  giving  a  list  of  them  with  enough 
explanations  In  the  Program  documentat I  on , 


7,4  Guide-lines  for  program  writing 

Tha  size  of  the  programs  developed  and  used  in  the 
htnd-eye  project  and  the  fact  that  tney  are  (and  were) 
written  by  a  number  of  different  people  over  a  long  period 
of  time,  makes  a  s#t  of  guide-lines  (If  not  enforced 
conventions)  for  orogram  writing  a  necessity. 

The  following  Is  a  long  list  of  parameters  that 
should  be  optimized,  or  at  least  considered,  when  writing  a 
proOrami 

(a)  Small  size  of  source  code. 

(b)  Small  size  of  object  code, 

(e)  Small  core  Image  at  exeoutlon  time. 

(d)  Fast  comp  I latlon, 

(e)  Fast  exeoutlon. 
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Readability  of  the  source  code  especially  oy 
other  users,  This  will  also  ease  modifying  the  code  when 
needed, 

*9)  Modularity!  General  utility  procedures  and 
others  should  be  written  so  that  they  can  be  cleanly  lifted 
out  of  one  Drogram  and  transplanted  Into  another,  This  means 
that  the  procedure  should  us*  formal  parameters  and  not 
global  variables  and  that  (t  should  make  explicit  reference 
to  any  subroutine  It  us*s  which  was  defined  outside  it. 

<h)  versatility?  The  procedures  should  be  applicable 
under  a  broad  range  of  conditions  without  failure  or 
r>otloeab|e  degradation,  Thay  should  not  be  too  versatile  if 
this  necessitates  complex  settings  of  flags  and  parameters, 

(I)  for  debugging  and  experimental  purposes#  the 
programs  need  conversational  ability  so  that  paramete-s  and 
even  structure  can  b«  modified  by  the  user  without 
recompilation,  However#  after  the  reliability  of  the  program 
has  been  satisfactorily  Droved#  most  of  this  code  should  be 
taken  out  In  an  orderly  fashion. 

Trying  to  optimize  all  the  above  parameters  I  =, 
working  at  cross  purposes,  One  trivial  example  The  use  o' 
a  table  Instead  of  computation  to  find  values  of  function? 
will  speed  up  execution  but  also  increase  needed  storage,  In 
the  programs  developed  In  this  research,  optimizing 
parameters  (f)  to  (I)  had  more  Importance  than  optimizinn 
parameters  (a)  to  (e),  I  would  like  to  have  a  compiler  with 
some  flexibility  as  to  what  parameters  It  tries  to  optimize. 

7,5  Module  organization 

Currently  we  have  about  ten  modules  apart  from 
drivers.  The  hand-eye  monitor  and  the  way  messages  are 
handled  allow  ccmplete  flexibility  as  to  which  module 
controls  what  module,  After  one  module  activates  a  nessege 
procedure  In  another  module  It  can  either  wait  till  the 
other  module  reDorts  termination  of  the  called  Procedure,  or 
go  ahead  and  continue  Independent  execution,  There  is  nc 
guaranty  th  t  the  called  procedure  wl | |  ever  return  control 
to  tha  calling  module, 

It  Is  up  to  the  programmer  to  build  some  structure 
in  the  control  of  modules  using  tha  flexibility  mentioned 
above,  Some  classes  of  structures  are  discussed  below, 

(a)  "Prussian  army'’  or  "Tree"  structure  (See  figure 

39(a)); 

Each  subtree  can  perform  a  specific  task,  The  taskg 
become  more  simple  and  specific  the  lower  they  are  i"  tnn 
tree,  Each  subtree  Is  "rugged"  in  the  sense  that  It  can 
perform  its  task  without  much  na|p  from  aDovo,  Command?  go 
down  the  tree  and  results  are  reported  upward.  Communication 
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between  modules  Is  done  only  through  c  common  ancestor  (or 
common  commander).  This  structure  Is  rigid  but  simple  and 
suitable  for  a  development  effort  which  does  not  have  to  be 
highly  coordinated,  The  development  can  start  from  both 
ends  of  the  tree  at  the  same  time,  The  rigidity  and 
simplicity  mak#  the  solution  of  the  optimization  of 
resources  allocation  very  difficult,  Another  disadv?  tage  is 
that  many  of  the  "soldiers”  have  very  similar  tasks  uUt  they 
cannot  be  used  outside  their  "units",  A  variant  of  this 
structure  is  discussed  later, 

(b)  ''Star"  structure  (See  Figure  39(b)): 

One  super-driver  Is  controlling  a|t  modules,  All  the 
communications  between  modules  are  done  through  the 
super-driver,  The  super-d  r  I  vej,  will  be  very  complex  but  can 
make  efficient  use  of  the  modules  and  other  resources,  A  I  | 
the  modules  have  to  be  designed  together,  otherwise  changes 
(sometimes  extensive),  w|||  be  needed  every  time  a  new 
module  Is  added  or  an  old  module  Is  changed,  since  for 
efflotent  planning  t^«  super-driver  needs  to  know  the  exact 
characteristics  of  the  modules, 

(c)  "Spoked  whael"  structure  (See  Figure  39(c)): 
Again  we  have  a  super-driver  controlling  all  modules,  But  in 
addition  there  are  channels  of  oommun . cat  I  on  and,  what  is 
more  'mportant,  channels  of  control  between  the  modules, 
governed  by  the  super-driver,  With  those  means  the  driver 
can  give  one  module  the  ability  to  control  other  modules  for 
the  purpose  of  achieving  a  certain  subgoal,  In  this  manner 
the  "knowledge"  needed  Is  diffused  but  not  to  the  extent  of 
the  tree  structure, 

The  organization  struoture  used  for  the  system 
described  here  Is  a  variant  of  the  tree  struoture,  where  the 
lower  level  modules  are  more  generalized  and  thus  can  be 
used  by  more  than  one  module,  (See  Figure  2),  Procedures 
In  the  generalized  lower  level  module  are  aotivated  by 
messages  sent  to  It  by  different  modules,  The  message 
handler  outs  these  messages  In  a  separate  queue  for  each 
module  to  wait  for  the  termination  of  all  activities 
required  by  prior  messages  In  the  queue,  Alternative  ways 
would  be  to  use  Identical  ooples  of  the  generalized  lower 
level  procedures,  or  to  make  them  reentrant, 

The  main  conclusion  to  Ds  drawn  from  this  chapter 
and  from  the  thesis  In  general  Is  that  things  are  always 
more  complicated  than  they  first  look,  in  fact,  the  entire 
field  of  Robotics  has  been  concerned  with  solving  problems 
which  appear  at  first  to  be  trivial, 
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□  MOOULE 


SUPER  MOOULE 


— COMMAND  (ONE  WAY)  AND  INFORMATION 
(BOTH  DIRECTIONS) 


fa)  "Prussian  Array" 

(b)  "Star" 

(c)  "Spoked  Wheel" 

Fig.  59- 

EXAMPLES  OF  MODULE  ORGANIZATION  STRUCTURE 

11^ 
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Appendix  A:  earner*  module 

A •  1  Camera  m0dule  message  procedures 

The  camera  module  Is  activated  by  the  following 
messages i 

CAM. I N I T 

Reads  the  camera  model  parameters  from  disc  file, 
CAM. UPDATE 

Readg  the  pan  tilt  and  foous  pots,  computes  the 
camera  transformation  (see  Seotlon  A, 2)  and  stores  it  in 
global  CAMERA^MODEL •  Since  the  pot  readings  are  noisy 

because  of  Power  supply  noise,  A/D  converter  noise,  high 

freauency  m«ohanlca!  vibrations  of  camera  parts  relative  to 

eaoh  other,  etc,,  the  pots  ar*  read  a  number  of  times  (40) 

and  the  readings  are  avePa$ed,  As  an  Indloatlon  of  the 
amount  of  noise  left  In  the  averaged  readings  we  use  the 
following  expression! 

<MAX-m:N)/C4*SQRT(N)3 

Where!  MAX  Is  the  highest  reading,  MIN  is  the  lowest 
reading,  and  n  Is  the  number  of  readings,  The  noise  Is 
ohecked  to  be  less  th&n  that  corresponding  to  1/20  of 
depth-of-f  I  e  |d  In  focus,  10M-4)  rads,  In  tilt  and 
2,5«10»(*4)  Pads, In  pan, 

CHNC.LENS(N) 

Rotates  the  turret  till  l«ns  no,  N  is  facing  the 
vldlcon,  It  takes  about  1  seo,  to  rotate  the  turret  to  the 
next  pos 1 1 1  on . 

MOVE. CAM(PAN, TILT) 

Moves  the  pan/tl|t  Mad  so  that  the  pan  and  tilt 
angles  In  radians  equal  to  PAN  and  TILT,  respectively,  The 
ve  |  oo I ty  of  the  tilt  movement  Is  ,06  rads/sec,  and  that  of 
pan  ,13  rads/sec, 

CHNC.FOCUS(RANC) 

Moves  the  vldlcon  to  focus  at  a  range  of  Rang  Inches 
from  the  lens  center,  The  velocity  of  the  vldlcon  movement 
Is  ,012  Inch/seo,  For  example*  to  change  the  distance  at 
whloh  the  camera  Is  focussed  from  20  inohes  to  50  inches 
with  the  2  Inch  fooal  length  lens  takes  11,5  sec, 


CAM.CENTER(N,X, Y,2) 


Computes  t ns  pan  and  tilt  angles  necessary  far  the 
principal  ray  of  the  camera  to  Pass  through  the  point 
<  X ,  V  •.  if )  In  t.c.s,  <  I  •  e  that  the  Image  of  this  point  will  br 
at  point  (PelfPP2)  |n  th«  l.c,s.>»  When  lens  no,  N  Is  used. 
(See  Section  A, 3),  It  also  computes  the  range  from  the  Ion,; 
oenter  to  the  point,  Then  the  camera  Is  servoed  to  the 
computed  setting  and  the  lens  Is  chanoed  accordingly. 

The  above  four  message  procedures  use  a  common  servn 
routine  In  the  following  ways  using  the  cot  constants  m.~ 
the  focus  eouatlon,  the  pan  and  tilt  angles  and  the  r  a  m  g  ,■ 
are  converted  to  pot  settings,  Tnese  settings  are  then 
checked  for  legality,  The  legality  of  the  lens  number  given 
to  It  is  also  checked,  Each  x/63  sec  the  readings  of  the 
pots  are  compared  to  those  required,  If  the  dlfferpnce<- 
are  larger  than  the  given  thresholds,  the  notors  ar- 
actuated  In  the  prooer  direction  and  run  at  the  constan- 
rateg  mentiored  above,  The  thresholds  (full)  correspond 
1/5  deptn-of -f I e  I  d  for  focus  and  3  mrads,  In  pan  and  tilt. 
Note  that  these  thresholds  are  much  higher  than  tha  noise 
requirements  mentioned  In  the  discussion  of  CAM. UPDATE,  Tne 
reason  Is  tnat  centering  Is  done  to  assure  that  tho  objec+ 
Mill  pe  aDo r o* 1  mate | y  In  the  middle  0f  the  field  of  vie*, 
where  the  performance  of  the  camera  and  the  model  are  oest. 
On  the  other  hand,  to  yet  oreclse  measurements  we  need  to 
know  where  the  camera  Is  pointing  with  high  precision. 
After  the  motion  Is  completed  or  aoorteo  CAf'-,IPDA’’L  is 
called  automat  I  cal  I y . 


CAM.PREDIN.X, Y,H) 

Computes  PAN,  TILT  and  RANG  as  in  CAMCEN^EP,  and 
then  computes  the  co r r espono i ng  camara  trans <  or  mat  I  on  and 
stores  it  in  global  CAMERA. PRED,  No  servolng  Is  done. 

The  output  of  the  camera  module  Is  as  follows  (nil 
global  variables)*  CAMFLC  will  be  nonzero  tf  an  error 
occurred  during  the  execution  of  the  Procedure  celled,  and 
Its  value  gives  the  cause  of  the  error,  CAMPAN,  CAMTIL  and 
CAMRANG  hold  the  current  Dan  and  tilt  angles  nnd  tne  range 
at  which  the  camera  Is  focussed,  CAMLENS  designates  the 
current  lens,  PaNPOT,  TIlPOT  and  FOCPOT  hold  the  current 
averaged  ladings  of  the  pots,  CAMERA.MODEL  and  CAMERa*PRE~ 
hold  the  transformation  of  the  current  camera  position  and 
of  the  position  mentioned  last  In  CAM.PRED , 

A, 2  Camera  transformation  matrix  computation 

Take  a  point  B  with  coordinates  CB3*(*t» yt, zt)  in 
the  t.c.s,  Its  coordinates  In  a  c.s,  parallel  to  the  t.c.s, 
and  translated  to  the  point  P  with  coordinates 
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CP3»CP1,P2,P3>  In  the  t,o.s,,  will  batCB3-CP3.  P  is  the 
point  of  Intersection  of  the  pan  and  tilt  axes.  Its 
coordlnatas  In  a  c,s,  rotatad  relative  to  the  last  one  with 
Euler  angles  PS  I  #  THETA  and  PH!  (right  hand  rotation  In  this 
order  >  will  pei 

C R1 <  PS  I #  THETA , PHI ) J*(CB3-CP3) . 

where  R1  Is  an  orthonor^al  rotation  matrix.  The  three  Euler 
angles  are  given  In  terms  of  our  conventions  for  the  pan  and 
tilt  angles  pan  and  TILT,  and  the  fixed  rotation  anoie  SWING 
as  ( see  Figure  26)1 

PSI«PAN*»J  THETA-T I LT I  PHI «SW I NG I 

We  will  denote  the  rotation  matrix  expressed  In  terms  of 
PAN,  TILT  and  SWING  by  CR23,  I, a; 

CR2(PAN,TILT,SWlNG)3«CRi(PSI,TETA,PHl)3. 

Hence  the  coordinates  of  the  point  B  In  the  last  coordinate 
system  will  pe;  C  R  2  3 • < CB3-CP3 ) ,  We  now  rotate  the  c.s.  with 
Euler  ang|es  -*/2,  0,  -»/2,  The  coordinates  of  the  point  B 
I n  the  new  c , s ,  will  be i 

[Rl<-*/2,0,-*/2>3*CR23<CB3-[P3) 

S a  define  a  matrix  CR3  by;  CR3bCRi(«»/2,0,»»/2) 3*CR23 , 
The  coordinates  of  a  point  B  In  the  lest  c.s,  will  be; 
C R 3 • < CB3-CP3) ,  Lastly,  we  translate  the  p.s.  to  a  ooint  C 
W|th  coordfnares  COP < CPI , DP2, 0P3 )  jn  the  |ast  c,s.  to  R®t 
^he  "camera  coordinate  system"  (c.c.s, ) ,  The  coordinates  of 
the  point  B  In  the  c,c,s,  which  will  be  denoted  by 
[U3S ( u»  v , w)  are: 

CU3S[R3*<C83-CP3)-CDP3 

For  the  s«Ke  of  completeness,  the  elements  of  the 
matrix  CRD  which  will  be  denoted  by  RIJ  <e,g,  R12)  will  be 
given  here  explicitly  In  terms  of  PAN,  TILT  and  SWING,  which 
will  be  abbreviated  by  P,T  and  S  respectively* 

R11=-C0S(S)*SIN(P)+SIN<S)«SIN<T)*C0S<P) 

R12=  COS(S)*COS(P)+SIN(S)»SIN(T)»SIN(P) 
R13=-SIN(S)»C0S(T) 

R2ls  SIN(S)eSlN(P)*COS(S)»SIN(T)*COS(P) 
R22*-SIN(S)*C0S(P)+C0S(S)*SIN(T)*SIN(P> 
R23=-C0S(S)*C0S(T) 

R3l=-C0S(T)*C0S(P) 

R32=-C0S(T)*SIN(P) 

R33=-SIN(T) 

The  Dan  and  tilt  angles  are  computed  from  the 
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corresponding  pot  readings  as  follows; 


PANspP0T2^PP0T0«PANP0T j 
TILT*TPOT0*TPOTD«TJLPOT) 

s,r‘l<»'  4.2,2  SHING  Is  an  Isv.rl.nt 


0«fln»  a  vactop  CNJ»(M1,M2,H3)  byi  CMJ  =  [R’.rp'*ro°;j. 

mitriv  rrSrn  ■■  #  ^  t  •  _ _ _  ’  * 


uorino  a  vector  [MJs(Hi,M2,M3>  by 
now  define  a  3*3  matrix  CEXT]  as  follows* 


EXUl  =  RU) 
EXT2l*R2l ) 
EXT3i=R3ll 


EXT12*R12|  EXT13*-M1j 
EXT22«R22;  EXT23*-M2) 
EXT32«R32|  EXT33*-M3j 


*h#  matrix  CEXT]  can  be  used  to  convert  coordinate* 
0"  t.oi.-too  from  t.c.s.  to  c.e.oT  In  thi 
following  manner*  For  a  point  B  on  the  table  with 
coord  nates  (xt,yt»fl)  form  a  vector  CT'J*(xt»yt»l> .  1 1  *5 

coordinates  In  the  c.c.s,  are  given  by*  CU]s[f XT3*CT] . 


To  convert  the  coordinates  of  a  point  In 
cenoted  by  Cu3«<u.vfw),  to  Its  eoordinate8  In  i.c.s. 
by  <X  .y|)  we  use  the  3*3  matrix  [  I N  t  j  defined  as 
(see  F igure  27) 1 


C  1  C  1  S  1 

denoted 

f 0  I  low* 


INT3isbi  1NT32=0» 


I  NT2iIpp2lfip  1/333 
INT33*1 | 


The  distance 
center  Is  computed 
f 0  I  lows  J 


batwean  the  vidlcon  fact  and  the  |ens 
from  the  reading  of  the  focus  oot  as 


FOC=FOCIEN0*FOCIENG*FOCPOT; 


The  matrix  C I  NT  3  Is  used 
compute  a  vector  C J3- < Jl, J2, j3 > 
xl«Jl/J3  and  yl=j2/j3j 


In  the  following  w  a  *  : 
by:  CJ3*CINTj#[ij]# 


-l  1  East,  Wfl  define  a  3*3  matrix  CCQl.]  =  [  !NT3*CEXT] 
This  matrix  incorporates  all  the  parameters  of  ’ 

model  and  its  use  should  be  odvIous  from  the 
'"''ll  repeat  !  t  anyway,  To 
on  a  table  given  by  Its 

to  Its  coordinates  In  ths  l,c,s,  <xi,y|),  we 

CT3s<xt,yt,l> ,  compute  CJ3=CCOL3*CT]f 

>I=J2/J3. 


the  camera 
above,  but  wr, 
concert  the  coordinates  of  a  ocirt 
coordinates  In  t.c.s,  t ( xt» yt, 2  ) 

form  a  victor 
and  x I s  j 1 / J  3 ; 


♦  0  a 
whose 


If  *e  want  to  comoute  t"e  image  point  cor rescond i ng 
point  no t  on  the  tah|e»top»  we  use  the  lens  center 
coordinates  In  the  t.c.s.  are  giver  by 
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lC]*CP3+INVC[R])»COP3,  We  compute  the  lint  through  the  given 
point  and  th#  lens  center,  than  we  compute  the  point  of 
■  ^before  00  b#tw##n  th  9  ,,n#  tnd  th#  table-top  end  proceed 

.  ,  compute  the  point  of  Intersection  between  tne 

tM  l.n,  o.nt.r  ind  Mn  ?«MSI 

ln  **•  . . form  th. 

'“"Put*  CJ].tNV<CC0LJ>.C!3,  tnd 

xtB Jl/ J3 »  yts J2/ j3, 


.  Tf*  f8 tr lx  C COL 3  Is  celled  the  camera  transformation 
T,tr  *•  It  is  computed  and  storec;  eaoh  time  when  the  cemere 
module  executes  CAM. UPDATE  or  CAM.PRED, 


A, 3  Derivation  of  camera  centering  equations 

d.  1°  ?#nttr  the  camera  on  a  oolnt  defined  by  Its 
coord  nates  Jn  the  t.c.s.  CB 3e ( xt. y t , Zt ) ,  we  compute  Its 
coordinates  |n  tne  l.c.s.i 

CU]=(u,v,w)j  CU3*CR3*(CP^-CB^)-CDP3l 
Then  we  compute  tho  needed  pan  and  tilt  angles  so  that 

*'«•"»  th.  ...uitm.  .ou.ti.n.  tSMt 

SWINCB0,  This  assumption  Is  Justified  since  tha 

calibrated  value  0f  SWING  Is  le89  than  2  deg,  with  thls 
assumption  w(  navel 

uB  <Pl-xt)«SlN(P)-(P2-yt)«C0S(P)-DPl 
vs-C(pl-xt)*CoS(p>+(p2-yt>*SIN(p)3*SIN(T)*(p3-zt)»CnS(T)-OP2 
w*  C<pl-xt)*COS<P)*(P2-yt)aSIN(P>3*C0S(T)m(P3-2t>*SlN(T)-DP3 


io I ut Ion 
:lom  v: 


by  PC). 
0,  and 


First  we  solve  u  =  0  for  P  <we  denote  the  s 
Then  we  substitute  this  value  Into  the  equation! 
solve  It  for  T  (we  denote  the  eolutlon  by  TO,  These  tw0 
equations  are  solved  by  converting  them  to  ouadratlc 
•ouatlons  In  COS(P)  and  COS(T)  respectively,  Last,  we 
substitute  PC  and  TC  Into  the  expression  for  w  to  get  the 
distance  between  the  len9  center  and  the  point  CRT  u. 
denote  this  distance  by  WC,  Then  we  use  the  focus  ecuatlon 
to  compute  the  corresponding  foo, 


A. 4  Derivation  of  pan/tMt  updating  eouatlons 

ihe  goal  of  the  pan/tllt  updating  routine  is  t„ 
update  or  recompute  the  pan  and  tilt  pots  oarameta™.  It  fj 
done  n  the  following  wayi  For  a  point  given  by  Its 

^SPJinaI*8  !?  th?  CBJ"(xt,yt, zt)  the  routine  finds 

the  corresponding  Image  coordinates  <xl,yl),  Fron  this 
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Information  we  would  Ilk#  to  oomoute  the 
of  the  camera.  These  angles  with  the 
readings  w|||  enable  us  to  recompute  the 


oan  and  tilt  angles 
cor  respond  I ng  rot 
oot  &aramete*s, 


we  deMnei  «*u/wj  fl«v/w 
Internal  mod#|  (9se  Section  A, 2) 


from  the  ecuatlons  of  the 
we  havei 


as  (  «  j -pPi >/( f oC»KX)J 

fJs(y|-pp2)/cfoc*KY)Mx!-pPl)/<333afoc*KY)  J 


n  *»3umOtlon  that  SWINGS,  we  get  from  thfc 
expressions  for  u,v  and  w  given  In  the  last  section  the 
following  two  eauatlons  which  we  have  to  solve  for  the  oan 
and  tilt  ang|«3  P  and  T « 


(Pl-xt)«SlN(P)-(P2-yt)*C0S(P)-DPls 

«MC(Pl-xt>«COS<P)*(P2-yt)»SIN{P>3#COS(T)*(P3-zt)»SlN(T>-OP3) 


«C  (Pl-xtleCOSfPUtPg- 
OMUPl-xtMCQStPJ+tP 


'tl*SIIJ(P>3«SIN(T)*(P3-2t)#CCS(T)-DP2  = 
!-yt)«S!MP)]«C0S(T)MP3-Zt).S:\(T)-DP3) 


fou,di  not  f,nd  an  ®  *  ®  I  I  c  i  t  analytic  solution  to 
the  above  eQuatlons.  but  the  following  approximate  solution 
proved  itself  to  be  aooroorlate, 


snive  ♦  de2°3*  ?y-PC  and  TC  thd  pan  and  tilt  angles  which 

solye  the  centering  equations  of  the  last  section  We  now 

cefln*  PC  and  TD  by:  PD=P-PC  and  TD»T.TC,  Set.  Jhtt  ?or 
Pw  and  TC  cre  actually  the  solutions  of  the  above 
•auatlons  and  henca  PO  and  TO  are  also  eaual  to  ?  In  thie 
case,  Since  In  the  undatlng  process  w§  centered  the  cannra 
be  .r*  finding  xi  and  y|  and  since  we  assume  that  the 
calibration  h#d  not  been  degraded  too  much,  we  can  assume 
that  u » v  <  w  and  hence  a.raCl  and  PD,TD<1  ori 


S I N ( o 0 ) =  P D |  SJN<  TO) =TD|  CCS ( PD ) =C0S ( TP ) = 1 | 

with  these  assumptions  we  get  the  following  eauatlons  for  ?' 

B no  TO  ! 


CG  +  «»DPl«COS(TC)>PU-Ca«DP23»TD-CaeDPl#SIN(TC)3*PD»TD  =  o.WC 
DPl*CSIN(Tc)-*-d«C0S(TC'3»PC"tWC*DP3*IJ*DP23«TD 

*Dpl#CCOS(Tc)-d»SIM(Tc)]«PC«’’D*'f»WC 


where  G  Is  defined  byi  G»<Pl-xt)»CjS(PC)*(P2-yt5 • S I \ ( ^ C ) . 

Dpi # Dp2 „ Cp3  are  of  the  same  order  of  WC  and  scme-hat 
smeller,  assuming  that  th#  camera  Is  not  looking  vertically 
oown  (which  Is  always  true  w|tn  the  camera  currently  used) 
we  can  say  the  same  about  G,  with  this  assumption  and  tn« 
assumptions  mentioned  above  we  can  simplify  the  ecuatlons  to 
read! 


G#PD 


aOD2»To  =  o«wC 


OPl» c  s I N(  TC)+0»COS<TC)  J)«PD-[WC«-DP3  J»TD*P*WC 

Solving  the  t*o  I  Inoar  oquatlono  and  using  the  sam« 
assumptions  again  wo  havoi 

P0»«»wc/C 

TOr-HC»Cd-«»Opl»S!N(TC)/C3/(HC+Op3) 
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Appendix  B:  Automatic  focus  module 

The  automatic  foousslng  routine  was  suggested  and 
experimentally  Implemented  by  J,M  TenenDaum,  end  then 
modified  by  me  for  use  In  the  foeus  pot  oallbratlon  updating 
program  and  similar  apolloatloni  (focussing  on  an  objeot  at 
unknown  rang#  s*en  as  a  "b|obH  In  the  Image  before  analyzing 
the  Image),  It  Is  too  orude,  as  It  I s »  to  find  range 
differences  between  two  points  oloser  than  1  Inoh  (at  30 
Inohee  the  deoth  of  field  of  the  2  Inoh  lens  Is  .6  Inoh), 

The  orlnolple  of  operation  Is  to  search  for  a  range 
by  moving  the  vldloon  relative  to  the  lens*  at  whloh  a  score 
function,  whloh  measures  the  the  Quality  of  focus*  Is 
max Iml zed, 

The  module  uses  a  window  defined  externally  In 
LOOK, AT,  Th#  block  diagram  ie  given  In  Figure  40,  Each  of 
the  blocks  exit  and  failure  Is  Indicated  (In  global  FOCFLG) 
If  either  INPUT  (see  Section  3,2,2)  or  the  camera  module 
have  failed.  Other  exit  oondltlone  are  mentlon«d  In  the 
blook  descriptions  whloh  follow, 

I  MIT 


The  camera  Is  fooussed  at  t’ie  ourrent  estimate  of 
the  range  (the  Initial  estimate  Is  given  as  a  parameter), 
SETCLIP  Is  celled  to  set  the  clips  to  braoket  the  Intensity 
range  seen  In  the  window  et  that  foous  setting. 

The  next  search  Interval  Is  selected  to  be  the 
larder  ofj  (#)  the  current  estimate  of  the  range  uncertainty 
(the  Initial  estimate  Is  given  as  a  parameter ), and  (b) 
3»D0FRE0  where  DOFREQ  Is  the  mlnlmlmum  attainable 
uncertainty  whloh  Is  defined  as  the  depth-of-f I e I d ,  OOFREQ 
Is  oomputed  by  an  aopro* Imatlon  to  a  more  complex  expression 
as  follow: 

DOFREQ* (R*2)*C*(Fi)/(F«2)  , 

R-ls  the  range  (the  ourrent  estimate  Is  used))  F • I s  the  lens 
focal  length)  F  * • I s  the  aperture  setting,  Plnce  the 
aperture  setting  Is  not  oomputer  controlled  It  Is  left  at 
maximum  opening  (1,4  for  a|l  lenses))  C-!s  the  dimension  of 
a  raster  c®|l  of  the  vldlcon  (apor x I  mate  I y  i/600  lrch),  For 
the  above  formula  w#  assumed  that  the  "circle  of  confusion" 
has  a  diameter  of  half  a  raster  unit, 

The  number  3  In  the  above  eauatlon  was  empirically 
chosen,  It  was  found  experimentally  that  the  criterion  used 
to  Judge  the  Quality  of  foousslng  (see  below)  has  values 
above  90X  of  the  maximum  In  an  Interval  of  3*D0FPEQ  around 
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start 


exit 


Fig.  ic. 


BLOCK-DIAGRAM  OF  THE  AUTOMATIC 


FOCUSSING 


MODI'LL 


IPU 


the  maximum, 

Tha  starch  Interval  Is  oentered  on  tha  ourrant  range 
•  t  t  •  W® t •  | 

MEASURE 


10  «quI d I etant  points  art  picked  In  the  search 
Interval,  At  eaoh  point  a  score  'unction  (criterion)  Is 
computed  by  TVSCRCEN  (see  below).  Tha  three  h|gheet  scores 
are  dynamically  maintained  with  their  corresponding  ranges, 

TVSCREEN 


■  .♦IhV,nd0W  *n  4  times  and  the  time-average 
Inteielty  Is  computed  at  eaoh  sampling  point,  Using  the 

averaged  Intensities,  the  magnitude  of  the  gradient  at  each 
sampling  point  (less  those  lying  on  the  perimeter  o'  the 
window)  Is  computed  using  the  following  approximation! 


C(X,Y)cSQRT(C!  CX-l,Y-l)*2ejCX-l,Y)*KX-l,Y*l, 
•KX«-l#Y-l)-2*I  <X*l,Y)-KX*l#y*l)3*2* 
CI(X-l.Y-l)*2e!<X|Y-l)M(X*l,Y-l) 

-I  <X-1#Y*1)-2#I (X, Y*1)-I(X*1,Y*1)3»2> 


where  Ilx.y)  Is  the  averaged  Intensity  at  the  sampling  point 
(x,y),  From  G(x,y>  we  oomoute  a  thresholded  value  of  the 
gradient's  magnitude  as  follows! 


H ( X »  Y ) ■  G ( X , Y  )  |f  G(X#Y)>CUT 
■  0  otherwise 

CUT  Is  a  parameter  fllven  to  the  program  or  computed  In  INIT 
to  be  a  fraction  of  the  maximum  of  H(X,Y)  over  the  window  at 
the  point  o'  highest  score,  The  soore  function  Is  the  sum  of 
H(XiY)  over  the  window. 


ACOMEVAl 


Using  the  3  highest  scores  and  the  corresponding 
ranges,  the  new  range  and  range-uncertainty  estimates  are 
computed,  Then  a  decision  Is  made  to  exit  or  enter  MEASURE 
again  according  to  the  following  rules! 

,  ,,  <■>,  If  «ll  the  scores  are  0  then  f COMEVAL  exits  a ng 
failure  Is  Indicated,  u 

(b>  If  the  range  corresponding  to  the  highest  score 
point  Is  too  close  to  the  search  Interval  boundary  points 
then  the  range  of  this  point  Is  the  new  estimate,  the  range 
uncertainty  estimate  Is  unchanged  and  MEASURE  Is  entered 
again, 

(c)  If  the  new  uncertainty  estimate  Is  larder  than 


125 


the  old  0n##  or  If  It  Is  smaller  than  DOFREQ,  ACO^EVAL 
exits, 

<d)  if  the  new  rang*  uncertainty  sstlmate  is  smaller 
than  the  old  one,  a  oounter  Is  incremented  (initially  set  to 
0),  If  the  countar  reads  lass  than  3  then  MEASURE  is  entered 
again  to  try  and  Improve  the  estimate#  otherwise  ACOMEVAl 
exits. 


The  automatic  focussing  module  Is  activated  by  the 
f o 1  lowing  messgej 

AUTOFOC<EST. RANGE#  RANG  I  fvIVAL.NCUT) 

Where'  EST_RANG*ls  the  Initial  range  estimate  (in  inches); 
RANGINVAL-Is  the  Initial  search  interval  (in  inches); 
NCUT-ls  the  value  for  out  |f  It  Is  positive,  or  a  f|ag  for 
automatic  setting  of  CUT  If  it  Is  negative, 


Tt?e  cutouts  of  the  module  are  the  following  giobals? 
fOCRANg-  Is  the  best  range  estimate)  pOCDELF-'s  the  best 
range  uncertainty  estlmatej  FOCFLG- I nd I  cates  success  or 
failure  of  the  module  and  the  source  of  failure, 


The  resolution  needed  of  the  focus  servo  is 
approximately  the  resolution  of  the  scanning  circuits  ( i , e 
the  dimension  of  a  raster  ceil)  and  Is  Independent  of  len-s 
focal  length  and  the  r«>nge  as  the  following  analysis  showt: 

Take  two  oolnts  at  distances  Xi  and  X2  from  the  |3ns 
center.  Denote  the  distances  between  lens  center  and  vidicon 
needed  to  focus  on  XI  and  X2,  Y1  and  Y2  respectively,  Usin-j 
the  focus  ecuation  we  have  apo rox I  mat® I y : 

Yl-Y2*(Ft2)*(X2-Xl)/(R»2) 

Wh#re  F  Is  the  lens  focal  length  and  Rs ( X1+X2 ) /? ,  If  we 
w#nt  10  oointe  In  #  search  Interval  of  at  !east  3*DCFRE0 
th#n  the  Increments  of  foe  between  two  consecutive  points 
will  be: 


DY=(rt2)*3*OOFREQ/C10*(R»2)D 


but  * 


UOFREQ=(R*2)»FMC/(Ft2) 
where  C  Is  the  scanning  resolution,  Hence: 

OY=3«F#*C/10 

Modifications  to  Improve  the  performance  of  the 
module  might  bs  of  two  typesl 
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(a)  Modification  of  th*  soora  funotton  computed  at 
aaeh  range,  One  possibility  Is  to  try  to  usa  the  concoct  of 
energy  of  high  spatial  fraquanolas  used  by  B,  Horn  at  m.I.t 
for  similar  applications  C113. 

(b)  Modification  of  tha  searoh  algorithm,  Here  we 
usa  the  three  highest  scorss  out  of  tan  computed  at 
equidistant  points  In  the  searoh  interval*  to  decide  whether 
to  stop  the  searoh  or  what  Interval  to  searoh  next.  If  the 
measurements  were  not  noisy  we  oou|d  use  a  Kind  of  binary 
searoh  whloh  Is  faster*  The  problem  of  finding  a  maximum  of 
a  funotlon  from  noisy  mesurements  of  its  values  belongs  to 
the  field  of  ’’stoohaetlo  approximations”,  Most  of  the 
theoretloal  results  of  this  field  are  ooncerned  with 
oonveroenee  properties  of  the  algorithms  after  a  large 
number  of  measurements  and  not  with  their  efficiency  [143. 
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